Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:12:11.312687 4165 test_util.cc:268] Using random seed: 1062894079
2024-05-03T02:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:11Z Disabled control of system clock
I20240503 02:12:11.351886 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--webserver_interface=127.4.17.126
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:12:11.713029 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:11.713392 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:11.713500 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:11.721325 4173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:11.721457 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:11.721503 4173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:11.721563 4173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:11.731315 4173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:11.732767 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:11.734730 4173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:11.747885 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.749181 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.750190 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:11.751408 4173 server_base.cc:1034] running on GCE node
I20240503 02:12:11.752441 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:11.754209 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:11.755506 4173 hybrid_clock.cc:648] HybridClock initialized: now 1714702331755419 us; error 54 us; skew 500 ppm
I20240503 02:12:11.755932 4173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:11.764714 4173 webserver.cc:460] Webserver started at http://127.4.17.126:35675/ using document root <none> and password file <none>
I20240503 02:12:11.765710 4173 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:11.765877 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:11.766239 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:11.769474 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/instance:
uuid: "00c621d795c44d0daa534261920d874a"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-ddxx"
I20240503 02:12:11.770404 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal/instance:
uuid: "00c621d795c44d0daa534261920d874a"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-ddxx"
I20240503 02:12:11.777232 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240503 02:12:11.781685 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:11.783730 4173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:12:11.784003 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "00c621d795c44d0daa534261920d874a"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-ddxx"
I20240503 02:12:11.784255 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:11.822937 4173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:11.823992 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:11.824344 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:11.844156 4173 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:43321
I20240503 02:12:11.844166 4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:43321 every 8 connection(s)
I20240503 02:12:11.845880 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:12:11.850378 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:11.853613 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4173
I20240503 02:12:11.853967 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal/instance
I20240503 02:12:11.857585 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:46221
--webserver_interface=127.4.17.125
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:12:11.859280 4239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:11.879273 4239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:12:11.880613 4190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
W20240503 02:12:11.882346 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:46221: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
I20240503 02:12:11.931795 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } attempt: 1
W20240503 02:12:11.933843 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:46221: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
I20240503 02:12:12.002235 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } attempt: 2
W20240503 02:12:12.004276 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:46221: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
I20240503 02:12:12.095674 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } attempt: 3
W20240503 02:12:12.097781 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:46221: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
W20240503 02:12:12.222472 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.222806 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.222908 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.230821 4241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:12.230957 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.231001 4241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:12.231060 4241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:12.240600 4241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.17.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.241075 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } attempt: 4
I20240503 02:12:12.242314 4241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20240503 02:12:12.243115 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:46221: Network error: Client connection negotiation failed: client connection to 127.4.17.125:46221: connect: Connection refused (error 111)
I20240503 02:12:12.244357 4241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:12.257174 4250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.258075 4249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.258916 4252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.260303 4241 server_base.cc:1034] running on GCE node
I20240503 02:12:12.261281 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.262854 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.264109 4241 hybrid_clock.cc:648] HybridClock initialized: now 1714702332264019 us; error 71 us; skew 500 ppm
I20240503 02:12:12.264510 4241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.267060 4241 webserver.cc:460] Webserver started at http://127.4.17.125:36335/ using document root <none> and password file <none>
I20240503 02:12:12.267820 4241 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.267953 4241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.268265 4241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:12.271117 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data/instance:
uuid: "df63329dc2b149d9b08cea341b540d1d"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.271957 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal/instance:
uuid: "df63329dc2b149d9b08cea341b540d1d"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.277603 4241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 02:12:12.281159 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.282577 4241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:12.282765 4241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal
uuid: "df63329dc2b149d9b08cea341b540d1d"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.282972 4241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.302704 4241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:12.303627 4241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.303946 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.322686 4241 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.125:46221
I20240503 02:12:12.322708 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.125:46221 every 8 connection(s)
I20240503 02:12:12.324175 4241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
I20240503 02:12:12.328322 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:12.330030 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4241
I20240503 02:12:12.330225 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-1/wal/instance
I20240503 02:12:12.331248 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:46689
--webserver_interface=127.4.17.124
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:12:12.334787 4310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:12.350970 4310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:12.362488 4310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:12:12.363720 4260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
W20240503 02:12:12.365288 4310 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.414621 4310 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 1
W20240503 02:12:12.416579 4310 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.484838 4310 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 2
W20240503 02:12:12.487053 4310 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.542416 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } attempt: 5
I20240503 02:12:12.550977 4239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:12:12.563653 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.578284 4310 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 3
W20240503 02:12:12.580122 4310 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.614965 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 1
W20240503 02:12:12.617486 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:12.685729 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 2
W20240503 02:12:12.687707 4239 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
W20240503 02:12:12.696601 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.696956 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.697103 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.704865 4312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:12.705050 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.705119 4312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:12.705214 4312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:12.714660 4312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.17.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.716104 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:12.718106 4312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:12.723409 4310 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 4
W20240503 02:12:12.725674 4310 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
W20240503 02:12:12.731478 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.732568 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.733525 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.734568 4312 server_base.cc:1034] running on GCE node
I20240503 02:12:12.735435 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.737035 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.738317 4312 hybrid_clock.cc:648] HybridClock initialized: now 1714702332738245 us; error 47 us; skew 500 ppm
I20240503 02:12:12.738708 4312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.741472 4312 webserver.cc:460] Webserver started at http://127.4.17.124:46343/ using document root <none> and password file <none>
I20240503 02:12:12.742265 4312 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.742409 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.742722 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:12.745707 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/instance:
uuid: "145afaa7548f4b7fafb23f20276f0cb4"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.746517 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal/instance:
uuid: "145afaa7548f4b7fafb23f20276f0cb4"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.752398 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:12:12.756222 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.757746 4312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:12.757942 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
uuid: "145afaa7548f4b7fafb23f20276f0cb4"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:12.758141 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.774185 4312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:12.775077 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.775400 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.793958 4312 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.124:46689
I20240503 02:12:12.793944 4239 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 3
I20240503 02:12:12.793960 4380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.124:46689 every 8 connection(s)
I20240503 02:12:12.795676 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
I20240503 02:12:12.799796 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:12.800429 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4312
I20240503 02:12:12.800626 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal/instance
I20240503 02:12:12.808090 4382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:12.814163 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap starting.
I20240503 02:12:12.818399 4239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:12.819924 4239 log.cc:824] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:12.829161 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: No bootstrap required, opened a new log
I20240503 02:12:12.834625 4382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:12.838169 4239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.838568 4239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:12.838696 4239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c621d795c44d0daa534261920d874a, State: Initialized, Role: FOLLOWER
I20240503 02:12:12.839475 4239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.841202 4388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.841907 4388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:12.841961 4239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:12.843895 4382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:12:12.856125 4382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Bootstrap starting.
I20240503 02:12:12.860000 4388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:12.860352 4382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:12.860347 4388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.861763 4382 log.cc:824] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:12.871344 4382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: No bootstrap required, opened a new log
I20240503 02:12:12.859867 4173 master_runner.cc:186] Error getting master registration for 127.4.17.125:46221: OK, instance_id { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" instance_seqno: 1714702332317231 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:12:12.872478 4173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:12:12.875641 4399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:12:12.875780 4399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:12:12.879817 4382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.880249 4382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:12.880380 4382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145afaa7548f4b7fafb23f20276f0cb4, State: Initialized, Role: FOLLOWER
I20240503 02:12:12.880645 4388 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:12.881168 4382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.882236 4402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.882580 4402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:12.883519 4382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:12.883543 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:12.884174 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c621d795c44d0daa534261920d874a in term 0.
I20240503 02:12:12.884860 4189 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c621d795c44d0daa534261920d874a, 145afaa7548f4b7fafb23f20276f0cb4; no voters:
I20240503 02:12:12.886015 4388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:12.886181 4388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:12.886292 4388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:12.890204 4388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.893764 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4"
I20240503 02:12:12.894009 4356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:12.897485 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c621d795c44d0daa534261920d874a in term 1.
I20240503 02:12:12.898029 4189 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c621d795c44d0daa534261920d874a, 145afaa7548f4b7fafb23f20276f0cb4; no voters:
I20240503 02:12:12.898732 4388 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 1 election: Requested vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:12.899245 4407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:12.899540 4407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 LEADER]: Becoming Leader. State: Replica: 00c621d795c44d0daa534261920d874a, State: Running, Role: LEADER
I20240503 02:12:12.900434 4407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:12.908900 4388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.909603 4388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:12.910799 4410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:12.913771 4410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:12.928367 4356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Refusing update from remote peer 00c621d795c44d0daa534261920d874a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:12.930794 4407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Connected to new peer: Peer: permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:12.941674 4312 master_runner.cc:186] Error getting master registration for 127.4.17.125:46221: OK, instance_id { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" instance_seqno: 1714702332317231 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:12:12.950783 4312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:12:12.954212 4402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.954528 4402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:12:12.960702 4417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:12:12.960848 4417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:12:12.964079 4407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.964401 4407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:12.966883 4410 catalog_manager.cc:1347] Generated new cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:12.967008 4410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:12.979173 4402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:12.979455 4402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:13.000744 4410 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:13.002621 4410 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:13.022642 4410 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Generated new TSK 0
I20240503 02:12:13.024935 4310 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } attempt: 5
I20240503 02:12:13.025439 4410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:13.037843 4310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Bootstrap starting.
I20240503 02:12:13.044162 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.65:0
--local_ip_for_outbound_sockets=127.4.17.65
--webserver_interface=127.4.17.65
--webserver_port=0
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:12:13.047314 4310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:13.051478 4310 log.cc:824] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:13.056195 4310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: No bootstrap required, opened a new log
I20240503 02:12:13.066624 4310 raft_consensus.cc:348] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.067104 4310 raft_consensus.cc:374] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:13.067258 4310 raft_consensus.cc:729] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df63329dc2b149d9b08cea341b540d1d, State: Initialized, Role: FOLLOWER
I20240503 02:12:13.068087 4310 consensus_queue.cc:260] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.081651 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:13.082172 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:13.084313 4310 sys_catalog.cc:564] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:13.084789 4283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:13.087515 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d"
I20240503 02:12:13.088693 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:13.093465 4283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Refusing update from remote peer 00c621d795c44d0daa534261920d874a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:13.094792 4388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:13.159893 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:13.160243 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:13.160643 4431 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:13.161000 4431 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:13.256949 4441 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:13.257074 4441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: loading cluster ID for follower catalog manager: success
I20240503 02:12:13.261199 4441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: acquiring CA information for follower catalog manager: success
I20240503 02:12:13.264180 4441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:12:13.524899 4443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:13.525283 4443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.528228 4214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:13.529784 4443 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:13.530114 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:13.530835 4332 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:13.531903 4443 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:12:13.559003 4423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:13.559437 4423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:13.559690 4423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:13.572813 4423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:13.573149 4423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.65
I20240503 02:12:13.588878 4423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.17.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.65
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:13.590731 4423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:13.593271 4423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:13.598127 4431 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:13.598402 4431 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.599893 4431 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:13.600793 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:13.601056 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df63329dc2b149d9b08cea341b540d1d in term 1.
I20240503 02:12:13.601059 4214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:13.601723 4260 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:13.602870 4431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:13.602993 4431 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:13.603082 4431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:13.606675 4431 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.607676 4431 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 2 election: Requested vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:13.608479 4214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a"
I20240503 02:12:13.609499 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4"
I20240503 02:12:13.610117 4260 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df63329dc2b149d9b08cea341b540d1d; no voters: 00c621d795c44d0daa534261920d874a, 145afaa7548f4b7fafb23f20276f0cb4
I20240503 02:12:13.610617 4431 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 02:12:13.618824 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.624862 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.627718 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:13.628849 4423 server_base.cc:1034] running on GCE node
I20240503 02:12:13.629791 4423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:13.631276 4423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:13.632618 4423 hybrid_clock.cc:648] HybridClock initialized: now 1714702333632533 us; error 63 us; skew 500 ppm
I20240503 02:12:13.633096 4423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:13.637538 4423 webserver.cc:460] Webserver started at http://127.4.17.65:45759/ using document root <none> and password file <none>
I20240503 02:12:13.638599 4423 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:13.638839 4423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:13.639258 4423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:13.643420 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data/instance:
uuid: "f4f9f079230a468f90e5e3a58f254b91"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-ddxx"
I20240503 02:12:13.644657 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/instance:
uuid: "f4f9f079230a468f90e5e3a58f254b91"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-ddxx"
I20240503 02:12:13.658607 4423 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20240503 02:12:13.662420 4285 raft_consensus.cc:1226] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 2 FOLLOWER]: Rejecting Update request from peer 00c621d795c44d0daa534261920d874a for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:13.667050 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.669118 4388 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:13.669127 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:13.669546 4423 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20240503 02:12:13.669726 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:13.669726 4388 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:13.669767 4423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal
uuid: "f4f9f079230a468f90e5e3a58f254b91"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-ddxx"
I20240503 02:12:13.669988 4388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c621d795c44d0daa534261920d874a, State: Running, Role: LEADER
I20240503 02:12:13.670065 4423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:13.670802 4388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:13.671716 4388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:12:13.676555 4461 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node 00c621d795c44d0daa534261920d874a not leader. Raft Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:13.710649 4423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:13.711848 4423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:13.712280 4423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:13.713589 4423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:13.715806 4423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:13.715906 4423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.716037 4423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:13.716105 4423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.751914 4423 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.65:39837
I20240503 02:12:13.751935 4573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.65:39837 every 8 connection(s)
I20240503 02:12:13.753763 4423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
I20240503 02:12:13.759627 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4423
I20240503 02:12:13.759886 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/instance
I20240503 02:12:13.803442 4576 heartbeater.cc:347] Connected to a master server at 127.4.17.125:46221
I20240503 02:12:13.803792 4576 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.804961 4576 heartbeater.cc:511] Master 127.4.17.125:46221 requested a full tablet report, sending...
I20240503 02:12:13.807952 4274 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:13.807979 4575 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:13.808128 4575 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.808495 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:13.808643 4574 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.808755 4575 heartbeater.cc:511] Master 127.4.17.126:43321 requested a full tablet report, sending...
I20240503 02:12:13.809238 4574 heartbeater.cc:511] Master 127.4.17.124:46689 requested a full tablet report, sending...
I20240503 02:12:13.810675 4203 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:13.810999 4346 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:13.878140 4399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:12:13.962996 4417 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:13.963096 4417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: loading cluster ID for follower catalog manager: success
I20240503 02:12:13.966037 4417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: acquiring CA information for follower catalog manager: success
I20240503 02:12:13.967988 4417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:12:14.056203 4460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:14.056404 4460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:14.057256 4460 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:14.057682 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:14.057837 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:14.057919 4285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c621d795c44d0daa534261920d874a in term 2.
I20240503 02:12:14.058336 4190 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:14.058610 4460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:12:14.058713 4460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:14.058825 4460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:14.061744 4460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:14.062512 4460 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 3 election: Requested vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:14.062783 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d"
I20240503 02:12:14.062997 4285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:14.062903 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4"
I20240503 02:12:14.065145 4285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c621d795c44d0daa534261920d874a in term 3.
I20240503 02:12:14.065538 4190 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d; no voters: 145afaa7548f4b7fafb23f20276f0cb4
I20240503 02:12:14.065927 4460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:14.066083 4460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 LEADER]: Becoming Leader. State: Replica: 00c621d795c44d0daa534261920d874a, State: Running, Role: LEADER
I20240503 02:12:14.066380 4460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:14.067612 4388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.067798 4388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:14.068027 4461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:14.069743 4461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:14.070622 4461 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:14.070717 4461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:14.071946 4461 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:14.072681 4461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Loaded TSK: 0
I20240503 02:12:14.073383 4461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:14.091158 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.66:0
--local_ip_for_outbound_sockets=127.4.17.66
--webserver_interface=127.4.17.66
--webserver_port=0
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:12:14.108598 4586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:14.108799 4586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:14.109843 4586 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:14.110397 4214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:14.110500 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:14.110770 4285 raft_consensus.cc:2354] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 145afaa7548f4b7fafb23f20276f0cb4 for earlier term 2. Current term is 3.
I20240503 02:12:14.111308 4332 leader_election.cc:398] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 145afaa7548f4b7fafb23f20276f0cb4 for earlier term 2. Current term is 3.
I20240503 02:12:14.111409 4332 leader_election.cc:401] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 02:12:14.111647 4586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 1 FOLLOWER]: Advancing to term 3
I20240503 02:12:14.114557 4586 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 145afaa7548f4b7fafb23f20276f0cb4 for earlier term 2. Current term is 3.
I20240503 02:12:14.314190 4203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.65:40709
W20240503 02:12:14.439797 4585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:14.440094 4585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:14.440301 4585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:14.448014 4585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:14.448236 4585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.66
I20240503 02:12:14.457655 4585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.17.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.66
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:14.459056 4585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:14.460947 4585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:14.466832 4356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Refusing update from remote peer 00c621d795c44d0daa534261920d874a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:12:14.467552 4388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Connected to new peer: Peer: permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:14.469594 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.469864 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:14.471731 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.472033 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:14.472307 4285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Refusing update from remote peer 00c621d795c44d0daa534261920d874a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:12:14.472956 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.473054 4460 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:14.473212 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:12:14.475700 4592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:14.475952 4602 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.476197 4602 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:14.476024 4599 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c621d795c44d0daa534261920d874a. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.476344 4599 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:12:14.476577 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.477506 4591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:14.478861 4585 server_base.cc:1034] running on GCE node
I20240503 02:12:14.479640 4585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:14.480713 4585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:14.481971 4585 hybrid_clock.cc:648] HybridClock initialized: now 1714702334481879 us; error 68 us; skew 500 ppm
I20240503 02:12:14.482338 4585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:14.485006 4585 webserver.cc:460] Webserver started at http://127.4.17.66:43927/ using document root <none> and password file <none>
I20240503 02:12:14.485811 4585 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:14.485957 4585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:14.486260 4585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:14.489336 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data/instance:
uuid: "1f50719d94564844b15e1a9211d48a81"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.490142 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/instance:
uuid: "1f50719d94564844b15e1a9211d48a81"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.495965 4585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:12:14.499675 4607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.501354 4585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:14.501533 4585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal
uuid: "1f50719d94564844b15e1a9211d48a81"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.501735 4585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:14.519970 4585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:14.520845 4585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:14.521191 4585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:14.522192 4585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:14.523763 4585 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:14.523835 4585 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.523947 4585 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:14.523996 4585 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.556216 4585 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.66:35355
I20240503 02:12:14.556262 4719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.66:35355 every 8 connection(s)
I20240503 02:12:14.557585 4585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
I20240503 02:12:14.559499 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4585
I20240503 02:12:14.559715 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/instance
I20240503 02:12:14.567171 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.67:0
--local_ip_for_outbound_sockets=127.4.17.67
--webserver_interface=127.4.17.67
--webserver_port=0
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:12:14.583824 4722 heartbeater.cc:347] Connected to a master server at 127.4.17.125:46221
I20240503 02:12:14.584285 4722 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:14.584483 4721 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:14.584640 4721 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:14.585842 4721 heartbeater.cc:511] Master 127.4.17.126:43321 requested a full tablet report, sending...
I20240503 02:12:14.587177 4722 heartbeater.cc:511] Master 127.4.17.125:46221 requested a full tablet report, sending...
I20240503 02:12:14.587275 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:14.587356 4720 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:14.587898 4720 heartbeater.cc:511] Master 127.4.17.124:46689 requested a full tablet report, sending...
I20240503 02:12:14.588274 4203 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:14.588649 4274 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:14.588873 4346 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:14.589347 4203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.66:58709
I20240503 02:12:14.816663 4575 heartbeater.cc:503] Master 127.4.17.126:43321 was elected leader, sending a full tablet report...
I20240503 02:12:14.898281 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "00c621d795c44d0daa534261920d874a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:14.898643 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: LEADER
W20240503 02:12:14.922405 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:14.922755 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:14.922912 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:14.930353 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:14.930584 4730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.67
I20240503 02:12:14.939447 4730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.17.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.67
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:14.940953 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:14.942747 4730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:14.955822 4739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.956622 4737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.957753 4736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:14.958896 4730 server_base.cc:1034] running on GCE node
I20240503 02:12:14.959558 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:14.960500 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:14.961724 4730 hybrid_clock.cc:648] HybridClock initialized: now 1714702334961659 us; error 40 us; skew 500 ppm
I20240503 02:12:14.962019 4730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:14.964394 4730 webserver.cc:460] Webserver started at http://127.4.17.67:39719/ using document root <none> and password file <none>
I20240503 02:12:14.965173 4730 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:14.965307 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:14.965595 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:14.968276 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data/instance:
uuid: "acc6d33969e74c209310005ae911c405"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.969197 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/instance:
uuid: "acc6d33969e74c209310005ae911c405"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.974617 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:14.978085 4745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.979501 4730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:14.979691 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal
uuid: "acc6d33969e74c209310005ae911c405"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-ddxx"
I20240503 02:12:14.979876 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:15.015133 4730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:15.016019 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:15.016332 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:15.017304 4730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:15.018932 4730 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:15.019011 4730 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.019145 4730 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:15.019201 4730 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.052488 4730 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.67:35993
I20240503 02:12:15.052536 4857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.67:35993 every 8 connection(s)
I20240503 02:12:15.053906 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
I20240503 02:12:15.063333 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4730
I20240503 02:12:15.063516 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/instance
I20240503 02:12:15.070200 4859 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:15.070552 4859 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:15.071836 4859 heartbeater.cc:511] Master 127.4.17.126:43321 requested a full tablet report, sending...
I20240503 02:12:15.074601 4203 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.075711 4203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.67:52535
I20240503 02:12:15.079089 4860 heartbeater.cc:347] Connected to a master server at 127.4.17.125:46221
I20240503 02:12:15.079253 4860 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:15.079596 4860 heartbeater.cc:511] Master 127.4.17.125:46221 requested a full tablet report, sending...
I20240503 02:12:15.079980 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:15.080099 4858 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:15.080354 4858 heartbeater.cc:511] Master 127.4.17.124:46689 requested a full tablet report, sending...
I20240503 02:12:15.080547 4274 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.081315 4346 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.091436 4721 heartbeater.cc:503] Master 127.4.17.126:43321 was elected leader, sending a full tablet report...
I20240503 02:12:15.095201 4165 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 02:12:15.125205 4203 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_fb826b4296744e3a92681d0112834b85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.125241 4202 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_ef647053879e4c5b9a6c529839490719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.125422 4204 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_94bb7b1f073d416ab9ab8757fb6c367e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.126837 4201 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_598376fb221c497ea72feda8c2a50ac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.127264 4202 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef647053879e4c5b9a6c529839490719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.127264 4203 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb826b4296744e3a92681d0112834b85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.127689 4201 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_598376fb221c497ea72feda8c2a50ac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.128207 4200 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_0bee8c4df2cc4b7d84d7fbfb1dd13714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.129055 4200 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bee8c4df2cc4b7d84d7fbfb1dd13714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.130843 4204 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94bb7b1f073d416ab9ab8757fb6c367e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.134964 4199 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_7967705ced17476f9383bba35b492bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.135056 4198 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_ccd4764e2caf4873ac83f0eabe14414e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.135879 4199 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7967705ced17476f9383bba35b492bd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.135962 4198 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccd4764e2caf4873ac83f0eabe14414e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.138471 4197 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_d22330b22e804239b0125eabd26ac99b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.139297 4197 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d22330b22e804239b0125eabd26ac99b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.142736 4196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_2aa235c1df9e431e8d21cbf89a5beff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.143383 4196 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aa235c1df9e431e8d21cbf89a5beff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.143949 4195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_9d5cdf6bb0b148d88b1211771704fc15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.144662 4195 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d5cdf6bb0b148d88b1211771704fc15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.169653 4202 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_fb826b4296744e3a92681d0112834b85" } new_table_name: "default.Table_e80a6bd8633b40718d4399a8149b69b9" modify_external_catalogs: true
I20240503 02:12:15.170442 4203 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_ef647053879e4c5b9a6c529839490719" } new_table_name: "default.Table_142ca54beeaf485c8667443452d2803c" modify_external_catalogs: true
I20240503 02:12:15.193387 4200 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_598376fb221c497ea72feda8c2a50ac5" } new_table_name: "default.Table_c24535208ac44ab8925d8bb9f586a326" modify_external_catalogs: true
I20240503 02:12:15.194655 4204 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_94bb7b1f073d416ab9ab8757fb6c367e" } new_table_name: "default.Table_b45447e6271449cb8b8d9936528cc41d" modify_external_catalogs: true
I20240503 02:12:15.195135 4201 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_0bee8c4df2cc4b7d84d7fbfb1dd13714" } new_table_name: "default.Table_2a744df7c5d946c0b7b49e940bee7966" modify_external_catalogs: true
I20240503 02:12:15.208057 4199 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_7967705ced17476f9383bba35b492bd2" } modify_external_catalogs: true
I20240503 02:12:15.208168 4198 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_d22330b22e804239b0125eabd26ac99b" } modify_external_catalogs: true
I20240503 02:12:15.208359 4199 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_7967705ced17476f9383bba35b492bd2" } modify_external_catalogs: true
I20240503 02:12:15.208474 4198 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_d22330b22e804239b0125eabd26ac99b" } modify_external_catalogs: true
I20240503 02:12:15.209409 4197 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_ccd4764e2caf4873ac83f0eabe14414e" } modify_external_catalogs: true
I20240503 02:12:15.209601 4197 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_ccd4764e2caf4873ac83f0eabe14414e" } modify_external_catalogs: true
W20240503 02:12:15.213022 4202 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43f7e89356e541d0b404f7f8bee00a6d has no consensus state
I20240503 02:12:15.213863 4202 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_9d5cdf6bb0b148d88b1211771704fc15" } modify_external_catalogs: true
I20240503 02:12:15.214124 4202 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_9d5cdf6bb0b148d88b1211771704fc15" } modify_external_catalogs: true
I20240503 02:12:15.213889 4195 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_2aa235c1df9e431e8d21cbf89a5beff8" } modify_external_catalogs: true
I20240503 02:12:15.217969 4195 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49522:
table { table_name: "default.Table_2aa235c1df9e431e8d21cbf89a5beff8" } modify_external_catalogs: true
W20240503 02:12:15.224691 4204 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11ae56645b97426b9458d9b1c91206c2 has no consensus state
W20240503 02:12:15.230687 4200 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet debf4d74181d4e97a7f46dd882d0b007 has no consensus state
W20240503 02:12:15.230722 4203 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cecaf785cb7e4aa1b6bef8d572e30706 consensus state has no leader
I20240503 02:12:15.239919 4197 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d94afdafd85b4d828e776caf67a2f8af
W20240503 02:12:15.239992 4201 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bd7febf6c46410199feb217351e80e7 has no consensus state
W20240503 02:12:15.244084 4190 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 43f7e89356e541d0b404f7f8bee00a6d consensus state has no leader
I20240503 02:12:15.247313 4199 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a9aa2dc3562e4e559c1a805b11e4b6df
I20240503 02:12:15.249825 4202 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fc01cde731544787a6f2f0323e5242ed
I20240503 02:12:15.250751 4198 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet e485dbb536784312946e85fac15242c2
I20240503 02:12:15.252136 4195 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fc8fc76c3b78499299caf0497f300059
I20240503 02:12:15.266913 4509 tablet_service.cc:1467] Processing CreateTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 (DEFAULT_TABLE table=default.Table_ef647053879e4c5b9a6c529839490719 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.268594 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecaf785cb7e4aa1b6bef8d572e30706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.272117 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4173
I20240503 02:12:15.273589 4234 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.277774 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d (DEFAULT_TABLE table=default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.279429 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f7e89356e541d0b404f7f8bee00a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.280403 4793 tablet_service.cc:1467] Processing CreateTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 (DEFAULT_TABLE table=default.Table_ef647053879e4c5b9a6c529839490719 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.280469 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae56645b97426b9458d9b1c91206c2 (DEFAULT_TABLE table=default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.280659 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7febf6c46410199feb217351e80e7 (DEFAULT_TABLE table=default.Table_2a744df7c5d946c0b7b49e940bee7966 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.281323 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae56645b97426b9458d9b1c91206c2 (DEFAULT_TABLE table=default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.281668 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae56645b97426b9458d9b1c91206c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.281724 4789 tablet_service.cc:1467] Processing CreateTablet for tablet debf4d74181d4e97a7f46dd882d0b007 (DEFAULT_TABLE table=default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.282236 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecaf785cb7e4aa1b6bef8d572e30706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.282509 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d (DEFAULT_TABLE table=default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.282991 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f7e89356e541d0b404f7f8bee00a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.283695 4651 tablet_service.cc:1467] Processing CreateTablet for tablet debf4d74181d4e97a7f46dd882d0b007 (DEFAULT_TABLE table=default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.284224 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debf4d74181d4e97a7f46dd882d0b007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.284659 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae56645b97426b9458d9b1c91206c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.284981 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7febf6c46410199feb217351e80e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.285833 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7febf6c46410199feb217351e80e7 (DEFAULT_TABLE table=default.Table_2a744df7c5d946c0b7b49e940bee7966 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.286303 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7febf6c46410199feb217351e80e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.289196 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debf4d74181d4e97a7f46dd882d0b007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.295483 4655 tablet_service.cc:1467] Processing CreateTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 (DEFAULT_TABLE table=default.Table_ef647053879e4c5b9a6c529839490719 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.296030 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecaf785cb7e4aa1b6bef8d572e30706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.303267 4923 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.306620 4922 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.307271 4923 tablet_bootstrap.cc:654] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.308634 4923 log.cc:824] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:15.310954 4922 tablet_bootstrap.cc:654] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.317291 4922 log.cc:824] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:15.319002 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d (DEFAULT_TABLE table=default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.319717 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f7e89356e541d0b404f7f8bee00a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.320955 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae56645b97426b9458d9b1c91206c2 (DEFAULT_TABLE table=default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.321597 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae56645b97426b9458d9b1c91206c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.323354 4505 tablet_service.cc:1467] Processing CreateTablet for tablet debf4d74181d4e97a7f46dd882d0b007 (DEFAULT_TABLE table=default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.323822 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debf4d74181d4e97a7f46dd882d0b007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.329413 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7febf6c46410199feb217351e80e7 (DEFAULT_TABLE table=default.Table_2a744df7c5d946c0b7b49e940bee7966 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.329954 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7febf6c46410199feb217351e80e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.337146 4923 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.337467 4923 ts_tablet_manager.cc:1367] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.012s
I20240503 02:12:15.355048 4923 raft_consensus.cc:348] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.357987 4923 raft_consensus.cc:374] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.358356 4923 raft_consensus.cc:729] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.360216 4923 consensus_queue.cc:260] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.362357 4926 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.363691 4923 ts_tablet_manager.cc:1398] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.026s user 0.008s sys 0.006s
I20240503 02:12:15.364290 4923 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.366150 4923 tablet_bootstrap.cc:654] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.372395 4926 tablet_bootstrap.cc:654] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.372416 4922 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.372771 4922 ts_tablet_manager.cc:1367] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.019s
I20240503 02:12:15.373715 4926 log.cc:824] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:15.377480 4923 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.377633 4923 ts_tablet_manager.cc:1367] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 02:12:15.378602 4923 raft_consensus.cc:348] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.378836 4923 raft_consensus.cc:374] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.378963 4923 raft_consensus.cc:729] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.379248 4923 consensus_queue.cc:260] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.381786 4923 ts_tablet_manager.cc:1398] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:15.383013 4926 tablet_bootstrap.cc:492] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.382755 4922 raft_consensus.cc:348] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.383170 4922 raft_consensus.cc:374] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.383266 4922 raft_consensus.cc:729] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.383327 4926 ts_tablet_manager.cc:1367] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240503 02:12:15.383770 4923 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.384032 4922 consensus_queue.cc:260] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.385558 4923 tablet_bootstrap.cc:654] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.393715 4922 ts_tablet_manager.cc:1398] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.021s user 0.015s sys 0.006s
I20240503 02:12:15.394006 4859 heartbeater.cc:503] Master 127.4.17.126:43321 was elected leader, sending a full tablet report...
I20240503 02:12:15.395009 4922 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.394879 4926 raft_consensus.cc:348] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.395326 4926 raft_consensus.cc:374] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.395421 4926 raft_consensus.cc:729] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.396214 4926 consensus_queue.cc:260] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.396672 4922 tablet_bootstrap.cc:654] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.401257 4922 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.401464 4922 ts_tablet_manager.cc:1367] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 02:12:15.404088 4922 raft_consensus.cc:348] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.405030 4922 raft_consensus.cc:374] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.405164 4922 raft_consensus.cc:729] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.405505 4922 consensus_queue.cc:260] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.406519 4923 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.406888 4923 ts_tablet_manager.cc:1367] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20240503 02:12:15.407744 4923 raft_consensus.cc:348] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.407872 4922 ts_tablet_manager.cc:1398] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:12:15.408046 4923 raft_consensus.cc:374] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.408202 4922 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.408195 4923 raft_consensus.cc:729] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.408538 4923 consensus_queue.cc:260] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.409562 4923 ts_tablet_manager.cc:1398] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:15.409956 4923 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.410451 4922 tablet_bootstrap.cc:654] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.411608 4923 tablet_bootstrap.cc:654] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.413235 4926 ts_tablet_manager.cc:1398] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.030s user 0.024s sys 0.005s
I20240503 02:12:15.413640 4926 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.417557 4933 raft_consensus.cc:482] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:15.417853 4933 raft_consensus.cc:504] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.419876 4923 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.420063 4923 ts_tablet_manager.cc:1367] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 02:12:15.420873 4923 raft_consensus.cc:348] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.421239 4923 raft_consensus.cc:374] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.421394 4923 raft_consensus.cc:729] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.421764 4926 tablet_bootstrap.cc:654] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.421707 4923 consensus_queue.cc:260] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.422519 4923 ts_tablet_manager.cc:1398] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:15.422857 4923 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.424460 4923 tablet_bootstrap.cc:654] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.432737 4923 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.433007 4923 ts_tablet_manager.cc:1367] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240503 02:12:15.433808 4923 raft_consensus.cc:348] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.434096 4923 raft_consensus.cc:374] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.434228 4923 raft_consensus.cc:729] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.434549 4923 consensus_queue.cc:260] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.436534 4923 ts_tablet_manager.cc:1398] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:15.437829 4933 leader_election.cc:288] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.440702 4922 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.440902 4922 ts_tablet_manager.cc:1367] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.002s
I20240503 02:12:15.441854 4922 raft_consensus.cc:348] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.442153 4922 raft_consensus.cc:374] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.442236 4922 raft_consensus.cc:729] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.447870 4922 consensus_queue.cc:260] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.448421 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debf4d74181d4e97a7f46dd882d0b007" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:15.449082 4922 ts_tablet_manager.cc:1398] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:12:15.449103 4813 raft_consensus.cc:2454] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:15.449414 4922 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.450636 4462 leader_election.cc:302] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:15.451054 4922 tablet_bootstrap.cc:654] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.452795 4926 tablet_bootstrap.cc:492] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.453431 4926 ts_tablet_manager.cc:1367] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.006s
I20240503 02:12:15.453217 4936 raft_consensus.cc:2781] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:15.453998 4936 raft_consensus.cc:482] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:15.454243 4936 raft_consensus.cc:3037] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.455914 4926 raft_consensus.cc:348] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.457597 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debf4d74181d4e97a7f46dd882d0b007" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:15.458256 4675 raft_consensus.cc:2454] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:15.461234 4926 raft_consensus.cc:374] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.461359 4926 raft_consensus.cc:729] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.461598 4926 consensus_queue.cc:260] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.462087 4922 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.462152 4926 ts_tablet_manager.cc:1398] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:12:15.462245 4922 ts_tablet_manager.cc:1367] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20240503 02:12:15.462849 4926 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.463101 4922 raft_consensus.cc:348] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.463292 4922 raft_consensus.cc:374] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.463387 4922 raft_consensus.cc:729] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.463601 4922 consensus_queue.cc:260] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.464170 4922 ts_tablet_manager.cc:1398] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:15.464468 4922 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.464514 4926 tablet_bootstrap.cc:654] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.466041 4922 tablet_bootstrap.cc:654] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.472316 4936 raft_consensus.cc:504] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.473600 4926 tablet_bootstrap.cc:492] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.473618 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debf4d74181d4e97a7f46dd882d0b007" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:15.473764 4926 ts_tablet_manager.cc:1367] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:12:15.473814 4675 raft_consensus.cc:3037] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.474534 4926 raft_consensus.cc:348] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.474733 4926 raft_consensus.cc:374] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.474792 4926 raft_consensus.cc:729] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.475036 4926 consensus_queue.cc:260] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.475550 4926 ts_tablet_manager.cc:1398] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:15.476806 4675 raft_consensus.cc:2454] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:15.477273 4926 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.480290 4462 leader_election.cc:302] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:15.480654 4933 raft_consensus.cc:2781] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:15.480959 4926 tablet_bootstrap.cc:654] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.483144 4922 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.483310 4922 ts_tablet_manager.cc:1367] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.011s
I20240503 02:12:15.484002 4922 raft_consensus.cc:348] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.484261 4922 raft_consensus.cc:374] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.484333 4922 raft_consensus.cc:729] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.484580 4922 consensus_queue.cc:260] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.485101 4922 ts_tablet_manager.cc:1398] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:15.489655 4936 leader_election.cc:288] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.490085 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debf4d74181d4e97a7f46dd882d0b007" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:15.490279 4813 raft_consensus.cc:3037] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.493572 4813 raft_consensus.cc:2454] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:15.494975 4933 raft_consensus.cc:686] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:15.495697 4933 consensus_queue.cc:237] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.500241 4926 tablet_bootstrap.cc:492] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.500399 4926 ts_tablet_manager.cc:1367] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20240503 02:12:15.501861 4926 raft_consensus.cc:348] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.502023 4926 raft_consensus.cc:374] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.502113 4926 raft_consensus.cc:729] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.502333 4926 consensus_queue.cc:260] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.502846 4926 ts_tablet_manager.cc:1398] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:15.503319 4926 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.505705 4926 tablet_bootstrap.cc:654] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.508422 4926 tablet_bootstrap.cc:492] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.508543 4926 ts_tablet_manager.cc:1367] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:15.509093 4926 raft_consensus.cc:348] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.509248 4926 raft_consensus.cc:374] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.509307 4926 raft_consensus.cc:729] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.509532 4926 consensus_queue.cc:260] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.509986 4926 ts_tablet_manager.cc:1398] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20240503 02:12:15.547098 4577 tablet.cc:2369] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:15.556588 4861 tablet.cc:2369] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:15.560204 4202 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_4d478de3b2a446dfb0181aa767598bec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.560892 4202 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d478de3b2a446dfb0181aa767598bec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.561347 4723 tablet.cc:2369] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:15.563731 4196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_f0be467f7bcf44989e1525c8831b1b58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.560777 4204 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_ee70ac9b02214ca68d4aa413db40c5be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.564246 4196 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0be467f7bcf44989e1525c8831b1b58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.564316 4204 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee70ac9b02214ca68d4aa413db40c5be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.570680 4196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_2f7c1f47295542bdaa0668b70d90dcf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.571070 4204 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_4ba3396bd3d040bebb3c9099309e615c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.571208 4196 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:15.571503 4204 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba3396bd3d040bebb3c9099309e615c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.579711 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (DEFAULT_TABLE table=default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.580186 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.581854 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (DEFAULT_TABLE table=default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.581962 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (DEFAULT_TABLE table=default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:15.582072 4204 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_db3a4a31a6ad4d81914cba26c1890d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:15.582264 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.582374 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:15.583608 4196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49522:
name: "default.Table_73fd503758ae407593561d3ad8412920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:15.584154 4196 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73fd503758ae407593561d3ad8412920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.585214 4203 master_service.cc:946] ReplaceTablet: received request to replace tablet 43f7e89356e541d0b404f7f8bee00a6d from {username='slave'} at 127.0.0.1:49522
I20240503 02:12:15.585320 4201 master_service.cc:946] ReplaceTablet: received request to replace tablet 11ae56645b97426b9458d9b1c91206c2 from {username='slave'} at 127.0.0.1:49522
I20240503 02:12:15.585618 4197 master_service.cc:946] ReplaceTablet: received request to replace tablet debf4d74181d4e97a7f46dd882d0b007 from {username='slave'} at 127.0.0.1:49522
I20240503 02:12:15.585848 4199 master_service.cc:946] ReplaceTablet: received request to replace tablet debf4d74181d4e97a7f46dd882d0b007 from {username='slave'} at 127.0.0.1:49522
I20240503 02:12:15.587353 4200 master_service.cc:946] ReplaceTablet: received request to replace tablet 1bd7febf6c46410199feb217351e80e7 from {username='slave'} at 127.0.0.1:49522
I20240503 02:12:15.588392 4922 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:15.590314 4922 tablet_bootstrap.cc:654] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:15.590332 4204 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db3a4a31a6ad4d81914cba26c1890d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:15.590785 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4173
I20240503 02:12:15.595254 4923 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:15.595803 4926 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:15.597126 4923 tablet_bootstrap.cc:654] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.597921 4926 tablet_bootstrap.cc:654] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:15.600224 4926 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:15.600358 4926 ts_tablet_manager.cc:1367] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:15.600950 4926 raft_consensus.cc:348] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.601148 4926 raft_consensus.cc:374] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.601253 4926 raft_consensus.cc:729] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.601482 4926 consensus_queue.cc:260] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.601886 4926 ts_tablet_manager.cc:1398] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:12:15.605577 4922 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:15.605738 4922 ts_tablet_manager.cc:1367] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.001s
I20240503 02:12:15.606470 4922 raft_consensus.cc:348] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.606631 4922 raft_consensus.cc:374] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.606720 4922 raft_consensus.cc:729] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.606988 4922 consensus_queue.cc:260] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.607542 4922 ts_tablet_manager.cc:1398] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:15.608963 4923 tablet_bootstrap.cc:492] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:15.609122 4923 ts_tablet_manager.cc:1367] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 02:12:15.612094 4923 raft_consensus.cc:348] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.612332 4923 raft_consensus.cc:374] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:15.612432 4923 raft_consensus.cc:729] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:15.612679 4923 consensus_queue.cc:260] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.613348 4923 ts_tablet_manager.cc:1398] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20240503 02:12:15.625474 4609 connection.cc:535] server connection from 127.0.0.1:57768 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:15.625526 4463 connection.cc:535] server connection from 127.0.0.1:51104 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:15.625979 4330 connection.cc:535] server connection from 127.0.0.1:52108 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:15.626343 4859 heartbeater.cc:644] Failed to heartbeat to 127.4.17.126:43321 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.17.126:43321 (error 108)
W20240503 02:12:15.626523 4721 heartbeater.cc:644] Failed to heartbeat to 127.4.17.126:43321 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.17.126:43321 (error 108)
W20240503 02:12:15.626914 4575 heartbeater.cc:644] Failed to heartbeat to 127.4.17.126:43321 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.17.126:43321 (error 108)
W20240503 02:12:15.627604 4261 connection.cc:535] server connection from 127.0.0.1:47844 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:15.627607 4865 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.126:43321)
W20240503 02:12:15.628489 4865 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.126:43321)
W20240503 02:12:15.629289 4865 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.126:43321)
W20240503 02:12:15.629962 4865 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.126:43321)
I20240503 02:12:15.698205 4927 raft_consensus.cc:482] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:15.698458 4927 raft_consensus.cc:504] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.700127 4927 leader_election.cc:288] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.707376 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b99fe59a9ab4f7ea06c64c24c3f45b3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:15.707367 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b99fe59a9ab4f7ea06c64c24c3f45b3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:15.707649 4813 raft_consensus.cc:2454] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:15.707831 4529 raft_consensus.cc:2454] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:15.708158 4608 leader_election.cc:302] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:15.708559 4927 raft_consensus.cc:2781] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:15.708694 4927 raft_consensus.cc:482] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:15.708793 4927 raft_consensus.cc:3037] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.710915 4927 raft_consensus.cc:504] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.711524 4927 leader_election.cc:288] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.712157 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b99fe59a9ab4f7ea06c64c24c3f45b3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:15.712146 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b99fe59a9ab4f7ea06c64c24c3f45b3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:15.712399 4813 raft_consensus.cc:3037] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.712376 4529 raft_consensus.cc:3037] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.715478 4529 raft_consensus.cc:2454] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:15.715445 4813 raft_consensus.cc:2454] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:15.715984 4608 leader_election.cc:302] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:15.716544 4927 raft_consensus.cc:2781] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:15.717336 4927 raft_consensus.cc:686] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:15.717996 4927 consensus_queue.cc:237] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.872479 4933 raft_consensus.cc:482] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:15.872687 4933 raft_consensus.cc:504] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.873307 4933 leader_election.cc:288] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.873862 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f7e89356e541d0b404f7f8bee00a6d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:15.873970 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f7e89356e541d0b404f7f8bee00a6d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:15.874097 4675 raft_consensus.cc:2454] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:15.874238 4813 raft_consensus.cc:2454] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:15.874545 4462 leader_election.cc:302] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:15.874845 4933 raft_consensus.cc:2781] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:15.874958 4933 raft_consensus.cc:482] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:15.875052 4933 raft_consensus.cc:3037] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.877512 4933 raft_consensus.cc:504] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.878086 4933 leader_election.cc:288] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.878563 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f7e89356e541d0b404f7f8bee00a6d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:15.878623 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f7e89356e541d0b404f7f8bee00a6d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:15.878809 4813 raft_consensus.cc:3037] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.878832 4675 raft_consensus.cc:3037] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.881651 4675 raft_consensus.cc:2454] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:15.881664 4813 raft_consensus.cc:2454] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:15.882086 4462 leader_election.cc:302] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:15.882380 4933 raft_consensus.cc:2781] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:15.882513 4933 raft_consensus.cc:686] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:15.882840 4933 consensus_queue.cc:237] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.989883 4927 raft_consensus.cc:482] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:15.990129 4927 raft_consensus.cc:504] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.990779 4927 leader_election.cc:288] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.991475 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecaf785cb7e4aa1b6bef8d572e30706" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:15.991539 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecaf785cb7e4aa1b6bef8d572e30706" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:15.991709 4813 raft_consensus.cc:2454] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:15.991801 4529 raft_consensus.cc:2454] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:15.992131 4608 leader_election.cc:302] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:15.992439 4927 raft_consensus.cc:2781] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:15.992542 4927 raft_consensus.cc:482] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:15.992638 4927 raft_consensus.cc:3037] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.994503 4927 raft_consensus.cc:504] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:15.995055 4927 leader_election.cc:288] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:15.995704 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecaf785cb7e4aa1b6bef8d572e30706" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:15.995780 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecaf785cb7e4aa1b6bef8d572e30706" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:15.995970 4529 raft_consensus.cc:3037] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.995986 4813 raft_consensus.cc:3037] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:15.998801 4529 raft_consensus.cc:2454] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:15.998801 4813 raft_consensus.cc:2454] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:15.999261 4608 leader_election.cc:302] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:15.999531 4927 raft_consensus.cc:2781] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:15.999740 4927 raft_consensus.cc:686] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:16.000048 4927 consensus_queue.cc:237] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.093600 4959 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:16.093660 4960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:16.093798 4959 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.093852 4960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.094566 4959 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:16.095312 4960 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:16.095559 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:16.095805 4283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 145afaa7548f4b7fafb23f20276f0cb4 in term 3.
I20240503 02:12:16.096053 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:16.096279 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df63329dc2b149d9b08cea341b540d1d in term 3.
W20240503 02:12:16.096345 4330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111)
I20240503 02:12:16.097115 4332 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:16.097420 4960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 02:12:16.097564 4960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 00c621d795c44d0daa534261920d874a)
I20240503 02:12:16.097571 4260 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:16.097682 4960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:12:16.097769 4259 leader_election.cc:334] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111)
I20240503 02:12:16.097898 4959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 02:12:16.098012 4959 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 00c621d795c44d0daa534261920d874a)
W20240503 02:12:16.098052 4330 leader_election.cc:334] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111)
I20240503 02:12:16.098146 4959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:16.100943 4959 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.100955 4960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.101667 4960 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 election: Requested vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:16.102007 4959 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 election: Requested vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:16.102296 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4"
I20240503 02:12:16.102567 4356 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df63329dc2b149d9b08cea341b540d1d in current term 4: Already voted for candidate 145afaa7548f4b7fafb23f20276f0cb4 in this term.
I20240503 02:12:16.102614 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d"
I20240503 02:12:16.102905 4283 raft_consensus.cc:2379] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 145afaa7548f4b7fafb23f20276f0cb4 in current term 4: Already voted for candidate df63329dc2b149d9b08cea341b540d1d in this term.
W20240503 02:12:16.103462 4259 leader_election.cc:334] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111)
I20240503 02:12:16.103592 4259 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df63329dc2b149d9b08cea341b540d1d; no voters: 00c621d795c44d0daa534261920d874a, 145afaa7548f4b7fafb23f20276f0cb4
W20240503 02:12:16.103736 4330 leader_election.cc:334] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111)
I20240503 02:12:16.103873 4959 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:12:16.103878 4330 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:16.104147 4960 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:12:16.152626 4933 raft_consensus.cc:482] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:16.152839 4933 raft_consensus.cc:504] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.153656 4933 leader_election.cc:288] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:16.154248 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ae56645b97426b9458d9b1c91206c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:16.154320 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ae56645b97426b9458d9b1c91206c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:16.154508 4675 raft_consensus.cc:2454] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:16.154528 4813 raft_consensus.cc:2454] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:16.154979 4462 leader_election.cc:302] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:16.155304 4933 raft_consensus.cc:2781] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:16.155392 4933 raft_consensus.cc:482] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:16.155489 4933 raft_consensus.cc:3037] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.157344 4933 raft_consensus.cc:504] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.157837 4933 leader_election.cc:288] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:16.158421 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ae56645b97426b9458d9b1c91206c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:16.158418 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ae56645b97426b9458d9b1c91206c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:16.158615 4813 raft_consensus.cc:3037] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.158635 4675 raft_consensus.cc:3037] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.160494 4675 raft_consensus.cc:2454] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:16.160563 4813 raft_consensus.cc:2454] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:16.160909 4462 leader_election.cc:302] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:16.161248 4933 raft_consensus.cc:2781] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:16.161437 4933 raft_consensus.cc:686] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:16.161960 4933 consensus_queue.cc:237] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.208758 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--webserver_interface=127.4.17.126
--webserver_port=35675
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:12:16.243577 4933 raft_consensus.cc:482] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:16.243800 4933 raft_consensus.cc:504] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.244570 4933 leader_election.cc:288] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:16.245112 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7febf6c46410199feb217351e80e7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:16.245189 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7febf6c46410199feb217351e80e7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:16.245414 4675 raft_consensus.cc:2454] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:16.245422 4813 raft_consensus.cc:2454] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:16.245851 4462 leader_election.cc:302] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:16.246117 4933 raft_consensus.cc:2781] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:16.246250 4933 raft_consensus.cc:482] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:16.246340 4933 raft_consensus.cc:3037] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.248222 4933 raft_consensus.cc:504] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.248798 4933 leader_election.cc:288] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:16.249281 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7febf6c46410199feb217351e80e7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:16.249503 4675 raft_consensus.cc:3037] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.249544 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7febf6c46410199feb217351e80e7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:16.249758 4813 raft_consensus.cc:3037] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.252388 4675 raft_consensus.cc:2454] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:16.252414 4813 raft_consensus.cc:2454] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:16.252882 4462 leader_election.cc:302] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:16.253238 4933 raft_consensus.cc:2781] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:16.253387 4933 raft_consensus.cc:686] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:16.253749 4933 consensus_queue.cc:237] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:16.504690 4927 consensus_queue.cc:1035] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.513183 4927 consensus_queue.cc:1035] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:16.577276 4967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:16.577626 4967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:16.577747 4967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:16.586331 4967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:16.586459 4967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:16.586530 4967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:16.586616 4967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:16.595992 4967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=35675
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:16.597506 4967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:16.599280 4967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:16.603374 4933 consensus_queue.cc:1035] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.615429 4933 consensus_queue.cc:1035] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:16.616981 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:16.617427 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:16.618562 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:16.619693 4967 server_base.cc:1034] running on GCE node
I20240503 02:12:16.621289 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:16.623210 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:16.624478 4967 hybrid_clock.cc:648] HybridClock initialized: now 1714702336624384 us; error 72 us; skew 500 ppm
I20240503 02:12:16.624795 4967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:16.627305 4967 webserver.cc:460] Webserver started at http://127.4.17.126:35675/ using document root <none> and password file <none>
I20240503 02:12:16.628022 4967 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:16.628147 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:16.634023 4967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:12:16.636950 4995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:16.638375 4967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:16.638543 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "00c621d795c44d0daa534261920d874a"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-ddxx"
I20240503 02:12:16.639391 4967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:16.661224 4967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:16.662123 4967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:16.662421 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:16.679275 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:16.679513 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.680368 5041 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:16.680946 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:16.681212 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df63329dc2b149d9b08cea341b540d1d in term 4.
I20240503 02:12:16.681710 4260 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:16.682226 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240503 02:12:16.682368 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:16.682474 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:12:16.683054 4967 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:43321
I20240503 02:12:16.683115 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:43321 every 8 connection(s)
I20240503 02:12:16.685223 4967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:12:16.685274 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.686481 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4"
I20240503 02:12:16.686692 4356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:12:16.689512 4356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df63329dc2b149d9b08cea341b540d1d in term 5.
I20240503 02:12:16.690524 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 4967
I20240503 02:12:16.692083 4260 leader_election.cc:302] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d; no voters:
I20240503 02:12:16.697446 5051 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:12:16.704210 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap starting.
I20240503 02:12:16.708704 5041 leader_election.cc:288] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [CANDIDATE]: Term 5 election: Requested vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:16.709897 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:12:16.710857 5041 raft_consensus.cc:686] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [term 5 LEADER]: Becoming Leader. State: Replica: df63329dc2b149d9b08cea341b540d1d, State: Running, Role: LEADER
I20240503 02:12:16.711726 5041 consensus_queue.cc:237] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.716820 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df63329dc2b149d9b08cea341b540d1d. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:16.717352 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:16.736642 5051 log.cc:824] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:16.770622 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=33 ignored=0} inserts{seen=33 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240503 02:12:16.771395 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap complete.
I20240503 02:12:16.780498 5051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.785645 5051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c621d795c44d0daa534261920d874a, State: Initialized, Role: FOLLOWER
I20240503 02:12:16.786478 5051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:16.787679 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:16.787979 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:16.789350 5051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:16.789400 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:16.789510 5021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df63329dc2b149d9b08cea341b540d1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a"
I20240503 02:12:16.789937 5021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 3 FOLLOWER]: Advancing to term 5
I20240503 02:12:16.794910 5021 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df63329dc2b149d9b08cea341b540d1d for term 5 because replica has last-logged OpId of term: 3 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 34.
I20240503 02:12:16.803287 5072 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:16.803547 5072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: loading cluster ID for follower catalog manager: success
I20240503 02:12:16.809818 5072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: acquiring CA information for follower catalog manager: success
I20240503 02:12:16.813776 5072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:12:16.938724 4933 consensus_queue.cc:1035] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.943586 4944 consensus_queue.cc:1035] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.949211 4933 consensus_queue.cc:1035] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.957832 4721 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:16.959475 4927 consensus_queue.cc:1035] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.964473 4859 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:16.964747 4859 heartbeater.cc:503] Master 127.4.17.126:43321 was elected leader, sending a full tablet report...
I20240503 02:12:16.965931 4575 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:16.970223 5012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "acc6d33969e74c209310005ae911c405" instance_seqno: 1714702335045023) as {username='slave'} at 127.4.17.67:41043; Asking this server to re-register.
I20240503 02:12:16.973073 4859 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:16.976402 5012 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:16.976888 5011 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1f50719d94564844b15e1a9211d48a81" instance_seqno: 1714702334548221) as {username='slave'} at 127.4.17.66:34023; Asking this server to re-register.
I20240503 02:12:16.977744 4721 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:16.982234 5011 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:16.982637 5012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" instance_seqno: 1714702333742995) as {username='slave'} at 127.4.17.65:44419; Asking this server to re-register.
I20240503 02:12:16.983211 4974 consensus_queue.cc:1035] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.989038 4944 consensus_queue.cc:1035] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:16.990010 4575 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:16.993392 5012 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:17.195709 4356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Refusing update from remote peer df63329dc2b149d9b08cea341b540d1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240503 02:12:17.196408 5055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:12:17.198482 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df63329dc2b149d9b08cea341b540d1d. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.198832 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:17.201020 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.201398 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:17.203400 5021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Refusing update from remote peer df63329dc2b149d9b08cea341b540d1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 5 index: 35. (term mismatch)
I20240503 02:12:17.203578 5021 pending_rounds.cc:76] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Aborting all ops after (but not including) 34
I20240503 02:12:17.203722 5021 pending_rounds.cc:97] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Aborting uncommitted WRITE_OP operation due to leader change: 3.35
I20240503 02:12:17.205663 5055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:12:17.206540 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.206776 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:17.209810 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.210242 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:17.210692 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df63329dc2b149d9b08cea341b540d1d. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.210875 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:17.227770 4944 consensus_queue.cc:1035] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:17.233283 4944 consensus_queue.cc:1035] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:17.249738 5057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:17.252642 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]
I20240503 02:12:17.253135 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_ee70ac9b02214ca68d4aa413db40c5be [id=1a2ffed6492747fba674919e859d7044]
I20240503 02:12:17.253494 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]
I20240503 02:12:17.254213 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_2a744df7c5d946c0b7b49e940bee7966 [id=8b3bd9d6487e473089474aa5cc736cf4]
I20240503 02:12:17.254557 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791]
I20240503 02:12:17.255038 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]
I20240503 02:12:17.255513 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]
I20240503 02:12:17.255813 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_142ca54beeaf485c8667443452d2803c [id=db26d21e021444d795aade9f022af228]
I20240503 02:12:17.256098 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_f0be467f7bcf44989e1525c8831b1b58 [id=e9ddc77e5a9c4944aeb87bfe64fe4b86]
I20240503 02:12:17.256383 5057 catalog_manager.cc:669] Loaded metadata for table default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6]
I20240503 02:12:17.258206 5057 tablet_loader.cc:96] loaded metadata for tablet 11ae56645b97426b9458d9b1c91206c2 (table default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791])
I20240503 02:12:17.258574 5057 tablet_loader.cc:96] loaded metadata for tablet 1b12e0c86f214f19b0bb11721cd83f29 (table default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6])
I20240503 02:12:17.258893 5057 tablet_loader.cc:96] loaded metadata for tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_2a744df7c5d946c0b7b49e940bee7966 [id=8b3bd9d6487e473089474aa5cc736cf4])
I20240503 02:12:17.259214 5057 tablet_loader.cc:96] loaded metadata for tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_e80a6bd8633b40718d4399a8149b69b9 [id=f1d32bef89ec4f2d83bc96623aef7ef6])
I20240503 02:12:17.259526 5057 tablet_loader.cc:96] loaded metadata for tablet 4d04dda695f044ebad1f4ec1bda9d75d (table default.Table_ee70ac9b02214ca68d4aa413db40c5be [id=1a2ffed6492747fba674919e859d7044])
I20240503 02:12:17.259819 5057 tablet_loader.cc:96] loaded metadata for tablet 5274ab7223e8424388bf5d742e5a3455 (table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31])
I20240503 02:12:17.260098 5057 tablet_loader.cc:96] loaded metadata for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (table default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda])
I20240503 02:12:17.260396 5057 tablet_loader.cc:96] loaded metadata for tablet a7fb095705c641ce929d9cc5029de3b9 (table default.Table_f0be467f7bcf44989e1525c8831b1b58 [id=e9ddc77e5a9c4944aeb87bfe64fe4b86])
I20240503 02:12:17.260814 5057 tablet_loader.cc:96] loaded metadata for tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_142ca54beeaf485c8667443452d2803c [id=db26d21e021444d795aade9f022af228])
I20240503 02:12:17.261265 5057 tablet_loader.cc:96] loaded metadata for tablet debf4d74181d4e97a7f46dd882d0b007 (table default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763])
I20240503 02:12:17.262015 5057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:17.263121 5057 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:17.263201 5057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:17.264847 5057 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:17.265764 5057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Loaded TSK: 0
I20240503 02:12:17.266624 5057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:17.287246 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:17.287613 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:17.303686 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 5274ab7223e8424388bf5d742e5a3455 (DEFAULT_TABLE table=default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.303715 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 5274ab7223e8424388bf5d742e5a3455 (DEFAULT_TABLE table=default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.304129 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274ab7223e8424388bf5d742e5a3455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.304160 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274ab7223e8424388bf5d742e5a3455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.304672 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 (DEFAULT_TABLE table=default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.305051 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b12e0c86f214f19b0bb11721cd83f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.305408 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 (DEFAULT_TABLE table=default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.305760 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b12e0c86f214f19b0bb11721cd83f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.311832 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 5274ab7223e8424388bf5d742e5a3455 (DEFAULT_TABLE table=default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.312350 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274ab7223e8424388bf5d742e5a3455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.313026 5103 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:17.314064 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 (DEFAULT_TABLE table=default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.314471 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b12e0c86f214f19b0bb11721cd83f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.315035 5103 tablet_bootstrap.cc:654] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.325695 5105 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.327731 5105 tablet_bootstrap.cc:654] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.329483 5106 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:17.329943 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4ba3396bd3d040bebb3c9099309e615c" } modify_external_catalogs: true
I20240503 02:12:17.330087 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2f7c1f47295542bdaa0668b70d90dcf2" } modify_external_catalogs: true
I20240503 02:12:17.330222 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4ba3396bd3d040bebb3c9099309e615c" } modify_external_catalogs: true
I20240503 02:12:17.330341 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2f7c1f47295542bdaa0668b70d90dcf2" } modify_external_catalogs: true
I20240503 02:12:17.330739 5103 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:17.330893 5103 ts_tablet_manager.cc:1367] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240503 02:12:17.331077 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4d478de3b2a446dfb0181aa767598bec" } modify_external_catalogs: true
I20240503 02:12:17.331279 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4d478de3b2a446dfb0181aa767598bec" } modify_external_catalogs: true
I20240503 02:12:17.331921 5103 raft_consensus.cc:348] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.332154 5103 raft_consensus.cc:374] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.332262 5103 raft_consensus.cc:729] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.332369 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ee70ac9b02214ca68d4aa413db40c5be" } modify_external_catalogs: true
I20240503 02:12:17.332562 5103 consensus_queue.cc:260] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.332607 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ee70ac9b02214ca68d4aa413db40c5be" } modify_external_catalogs: true
I20240503 02:12:17.332656 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f0be467f7bcf44989e1525c8831b1b58" } new_table_name: "default.Table_c0b1f5e771194ca79a9b98901d09ee64" modify_external_catalogs: true
I20240503 02:12:17.333055 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_142ca54beeaf485c8667443452d2803c" } new_table_name: "default.Table_973e6bc717c34997b7ab76b30f3d64e4" modify_external_catalogs: true
I20240503 02:12:17.333228 5103 ts_tablet_manager.cc:1398] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:12:17.333351 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c24535208ac44ab8925d8bb9f586a326" } modify_external_catalogs: true
I20240503 02:12:17.333566 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c24535208ac44ab8925d8bb9f586a326" } modify_external_catalogs: true
I20240503 02:12:17.334779 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet debf4d74181d4e97a7f46dd882d0b007 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.335875 5103 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:17.337509 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e80a6bd8633b40718d4399a8149b69b9" } new_table_name: "default.Table_75654fad5307429586e50445bc45687f" modify_external_catalogs: true
I20240503 02:12:17.337656 5103 tablet_bootstrap.cc:654] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.340945 5103 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:17.341107 5103 ts_tablet_manager.cc:1367] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:17.341895 5103 raft_consensus.cc:348] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.342116 5105 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.342111 5103 raft_consensus.cc:374] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.342201 5103 raft_consensus.cc:729] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.342253 5105 ts_tablet_manager.cc:1367] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 02:12:17.342458 5103 consensus_queue.cc:260] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.342949 5103 ts_tablet_manager.cc:1398] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:17.343137 5105 raft_consensus.cc:348] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.343322 5105 raft_consensus.cc:374] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.343403 5105 raft_consensus.cc:729] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.343631 5105 consensus_queue.cc:260] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.347242 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2a744df7c5d946c0b7b49e940bee7966" } new_table_name: "default.Table_91dc254b83f74fcaa5e18ac066681ce6" modify_external_catalogs: true
I20240503 02:12:17.348481 5105 ts_tablet_manager.cc:1398] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:17.350534 5105 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.352095 5105 tablet_bootstrap.cc:654] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.357993 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1b12e0c86f214f19b0bb11721cd83f29
I20240503 02:12:17.358286 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3
I20240503 02:12:17.359724 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.359838 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.359846 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.360342 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.360836 5113 tablet_replica.cc:330] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.361073 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.361429 5113 raft_consensus.cc:2229] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.361845 5113 raft_consensus.cc:2258] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.364509 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.365097 5106 tablet_bootstrap.cc:654] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.369953 5105 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.370188 5105 ts_tablet_manager.cc:1367] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20240503 02:12:17.370394 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.370676 5115 tablet_replica.cc:330] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.370985 5105 raft_consensus.cc:348] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.371177 5115 raft_consensus.cc:2229] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:17.371234 5105 raft_consensus.cc:374] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.371300 5105 raft_consensus.cc:729] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.371778 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 5274ab7223e8424388bf5d742e5a3455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.372001 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 5274ab7223e8424388bf5d742e5a3455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.372054 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4d04dda695f044ebad1f4ec1bda9d75d
I20240503 02:12:17.372447 5115 raft_consensus.cc:2258] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.373064 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d04dda695f044ebad1f4ec1bda9d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.373112 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d04dda695f044ebad1f4ec1bda9d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.372937 5105 consensus_queue.cc:260] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.373776 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.373996 5114 tablet_replica.cc:330] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.374482 5114 raft_consensus.cc:2229] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.373409 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 5274ab7223e8424388bf5d742e5a3455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.373586 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d04dda695f044ebad1f4ec1bda9d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.375137 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.375384 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.376685 5105 ts_tablet_manager.cc:1398] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:12:17.377959 5106 tablet_bootstrap.cc:492] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:17.378116 5106 ts_tablet_manager.cc:1367] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20240503 02:12:17.378716 5115 ts_tablet_manager.cc:1875] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.376279 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.379619 5114 raft_consensus.cc:2258] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.378999 5106 raft_consensus.cc:348] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.380244 5106 raft_consensus.cc:374] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.380343 5106 raft_consensus.cc:729] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.380600 5106 consensus_queue.cc:260] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.383258 5106 ts_tablet_manager.cc:1398] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:17.383584 5106 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:17.385732 5114 ts_tablet_manager.cc:1875] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:17.389374 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43f7e89356e541d0b404f7f8bee00a6d consensus state has no leader
I20240503 02:12:17.390393 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet debf4d74181d4e97a7f46dd882d0b007 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:17.390722 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cecaf785cb7e4aa1b6bef8d572e30706 consensus state has no leader
I20240503 02:12:17.390988 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 11ae56645b97426b9458d9b1c91206c2 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:17.391302 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7fb095705c641ce929d9cc5029de3b9 consensus state has no leader
I20240503 02:12:17.391556 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 1bd7febf6c46410199feb217351e80e7 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.391880 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 43f7e89356e541d0b404f7f8bee00a6d from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.395085 5106 tablet_bootstrap.cc:654] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.400405 5115 ts_tablet_manager.cc:1888] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.400633 5115 log.cc:1196] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/6b99fe59a9ab4f7ea06c64c24c3f45b3
I20240503 02:12:17.401352 5115 ts_tablet_manager.cc:1909] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:17.402374 5114 ts_tablet_manager.cc:1888] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.402539 5114 log.cc:1196] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/6b99fe59a9ab4f7ea06c64c24c3f45b3
I20240503 02:12:17.403039 5114 ts_tablet_manager.cc:1909] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:17.403347 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bd7febf6c46410199feb217351e80e7 consensus state has no leader
I20240503 02:12:17.404992 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (table default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]) successfully deleted
W20240503 02:12:17.406291 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1b12e0c86f214f19b0bb11721cd83f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b12e0c86f214f19b0bb11721cd83f29 already in progress: creating tablet
I20240503 02:12:17.406590 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (table default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]) successfully deleted
I20240503 02:12:17.406963 5115 tablet_replica.cc:330] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.407464 5114 tablet_replica.cc:330] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.407776 5115 raft_consensus.cc:2229] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.407929 5115 raft_consensus.cc:2258] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.408488 5106 tablet_bootstrap.cc:492] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:17.408655 5115 ts_tablet_manager.cc:1875] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.409060 5106 ts_tablet_manager.cc:1367] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.007s
I20240503 02:12:17.409880 5106 raft_consensus.cc:348] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.410125 5106 raft_consensus.cc:374] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.410220 5106 raft_consensus.cc:729] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.410679 5106 consensus_queue.cc:260] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.413507 5106 ts_tablet_manager.cc:1398] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:17.413935 5114 raft_consensus.cc:2229] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.414392 5114 raft_consensus.cc:2258] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.415330 5114 ts_tablet_manager.cc:1875] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.417800 5113 ts_tablet_manager.cc:1875] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.419133 5115 ts_tablet_manager.cc:1888] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:17.419195 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 43f7e89356e541d0b404f7f8bee00a6d consensus state has no leader
I20240503 02:12:17.419664 5115 log.cc:1196] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.420413 5115 ts_tablet_manager.cc:1909] T 5274ab7223e8424388bf5d742e5a3455 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:17.422048 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 5274ab7223e8424388bf5d742e5a3455 (table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]) successfully deleted
I20240503 02:12:17.422396 5114 ts_tablet_manager.cc:1888] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:17.422549 5114 log.cc:1196] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/1b12e0c86f214f19b0bb11721cd83f29
I20240503 02:12:17.422912 5114 ts_tablet_manager.cc:1909] T 1b12e0c86f214f19b0bb11721cd83f29 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:17.423086 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4d04dda695f044ebad1f4ec1bda9d75d because the tablet was not found. No further retry: Not found: Tablet not found: 4d04dda695f044ebad1f4ec1bda9d75d
I20240503 02:12:17.423542 5115 tablet_replica.cc:330] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.423965 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 1b12e0c86f214f19b0bb11721cd83f29 (table default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]) successfully deleted
I20240503 02:12:17.423975 5115 raft_consensus.cc:2229] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.424233 5115 raft_consensus.cc:2258] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.424309 5114 tablet_replica.cc:330] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.424789 5114 raft_consensus.cc:2229] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.424983 5114 raft_consensus.cc:2258] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.425594 5115 ts_tablet_manager.cc:1875] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.426860 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_db3a4a31a6ad4d81914cba26c1890d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.427167 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 5274ab7223e8424388bf5d742e5a3455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.427354 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3e37722f22364e62b5e10db841f1b963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.427811 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db3a4a31a6ad4d81914cba26c1890d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.427946 4722 heartbeater.cc:503] Master 127.4.17.125:46221 was elected leader, sending a full tablet report...
I20240503 02:12:17.429803 5114 ts_tablet_manager.cc:1875] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.430684 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1564b67cd15c49349918700f41524b71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.430824 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_58a13c96da8f4ca089be3162a5b53abb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.431174 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1564b67cd15c49349918700f41524b71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:17.431335 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58a13c96da8f4ca089be3162a5b53abb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.431646 5113 ts_tablet_manager.cc:1888] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.431895 4860 heartbeater.cc:503] Master 127.4.17.125:46221 was elected leader, sending a full tablet report...
I20240503 02:12:17.431922 5113 log.cc:1196] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/6b99fe59a9ab4f7ea06c64c24c3f45b3
I20240503 02:12:17.432886 5113 ts_tablet_manager.cc:1909] T 6b99fe59a9ab4f7ea06c64c24c3f45b3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.434264 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_73fd503758ae407593561d3ad8412920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.434862 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73fd503758ae407593561d3ad8412920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.436182 5113 tablet_replica.cc:330] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.436381 5113 raft_consensus.cc:2229] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.436533 5113 raft_consensus.cc:2258] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.437166 5113 ts_tablet_manager.cc:1875] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.437419 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 (table default.Table_4d478de3b2a446dfb0181aa767598bec [id=369ecd52371a40e489534627b91eacda]) successfully deleted
I20240503 02:12:17.440802 5114 ts_tablet_manager.cc:1888] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:17.440984 5114 log.cc:1196] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.441373 5114 ts_tablet_manager.cc:1909] T 5274ab7223e8424388bf5d742e5a3455 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:17.441905 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 43f7e89356e541d0b404f7f8bee00a6d
I20240503 02:12:17.442446 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 43f7e89356e541d0b404f7f8bee00a6d deleted and replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:17.442790 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b45447e6271449cb8b8d9936528cc41d" } new_table_name: "default.Table_94169f163e904089be031c1def8b5ba6" modify_external_catalogs: true
I20240503 02:12:17.444144 4970 raft_consensus.cc:482] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:17.444363 5115 ts_tablet_manager.cc:1888] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.444487 5115 log.cc:1196] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.444315 4970 raft_consensus.cc:504] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:17.444873 5115 ts_tablet_manager.cc:1909] T debf4d74181d4e97a7f46dd882d0b007 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:17.445232 5113 ts_tablet_manager.cc:1888] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:17.445369 5113 log.cc:1196] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1b12e0c86f214f19b0bb11721cd83f29
I20240503 02:12:17.445711 5113 ts_tablet_manager.cc:1909] T 1b12e0c86f214f19b0bb11721cd83f29 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.446595 4970 leader_election.cc:288] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:17.446913 5114 tablet_replica.cc:330] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.446821 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_aecae221afdc4f5ebe0462800c22c19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.447146 5114 raft_consensus.cc:2229] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:17.447471 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aecae221afdc4f5ebe0462800c22c19f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.447649 5114 raft_consensus.cc:2258] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.447784 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet debf4d74181d4e97a7f46dd882d0b007 (table default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]) successfully deleted
W20240503 02:12:17.448052 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5274ab7223e8424388bf5d742e5a3455 because the tablet was not found. No further retry: Not found: Tablet not found: 5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.448299 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 5274ab7223e8424388bf5d742e5a3455 (table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]) successfully deleted
W20240503 02:12:17.448588 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4d04dda695f044ebad1f4ec1bda9d75d because the tablet was not found. No further retry: Not found: Tablet not found: 4d04dda695f044ebad1f4ec1bda9d75d
I20240503 02:12:17.448956 4576 heartbeater.cc:503] Master 127.4.17.125:46221 was elected leader, sending a full tablet report...
I20240503 02:12:17.449293 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.449306 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.449509 5115 tablet_replica.cc:330] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.449712 5115 raft_consensus.cc:2229] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.449862 5115 raft_consensus.cc:2258] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.449859 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.450973 4272 catalog_manager.cc:5484] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:17.451923 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_973e6bc717c34997b7ab76b30f3d64e4 [id=db26d21e021444d795aade9f022af228]). Expected version 2 got 0
I20240503 02:12:17.452306 4272 catalog_manager.cc:5484] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } }
I20240503 02:12:17.452605 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 2 got 0
I20240503 02:12:17.452800 4272 catalog_manager.cc:5484] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } }
I20240503 02:12:17.453071 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 11ae56645b97426b9458d9b1c91206c2 (table default.Table_b45447e6271449cb8b8d9936528cc41d [id=a9a1676f5f6a41fb9d7614f6c9796791]). Expected version 1 got 0
W20240503 02:12:17.427812 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e37722f22364e62b5e10db841f1b963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.455765 5115 ts_tablet_manager.cc:1875] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.457177 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_06755c09ee504c3d8469e8b4d2acf8a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.457711 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06755c09ee504c3d8469e8b4d2acf8a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.457824 5114 ts_tablet_manager.cc:1875] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.459741 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_97eb124b7de24c8893a1c70ec2897e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.460332 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97eb124b7de24c8893a1c70ec2897e7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.468341 5113 tablet_replica.cc:330] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.468958 5113 raft_consensus.cc:2229] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.469637 5113 raft_consensus.cc:2258] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.471386 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1b12e0c86f214f19b0bb11721cd83f29 (table default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]) successfully deleted
I20240503 02:12:17.473996 5113 ts_tablet_manager.cc:1875] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.478879 5115 ts_tablet_manager.cc:1888] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.479053 5115 log.cc:1196] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/43f7e89356e541d0b404f7f8bee00a6d
I20240503 02:12:17.479508 5115 ts_tablet_manager.cc:1909] T 43f7e89356e541d0b404f7f8bee00a6d P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:17.480762 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
I20240503 02:12:17.483573 5114 ts_tablet_manager.cc:1888] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.483760 5114 log.cc:1196] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.484124 5113 ts_tablet_manager.cc:1888] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:17.484248 5114 ts_tablet_manager.cc:1909] T debf4d74181d4e97a7f46dd882d0b007 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:17.484407 5113 log.cc:1196] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.485236 5113 ts_tablet_manager.cc:1909] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.485792 5114 tablet_replica.cc:330] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.485774 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet debf4d74181d4e97a7f46dd882d0b007 (table default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]) successfully deleted
I20240503 02:12:17.485987 5114 raft_consensus.cc:2229] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:17.486207 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 5274ab7223e8424388bf5d742e5a3455 (table default.Table_2f7c1f47295542bdaa0668b70d90dcf2 [id=cc5457075695470790333ce116111b31]) successfully deleted
I20240503 02:12:17.486366 5114 raft_consensus.cc:2258] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.487094 5113 tablet_replica.cc:330] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:17.487196 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4d04dda695f044ebad1f4ec1bda9d75d because the tablet was not found. No further retry: Not found: Tablet not found: 4d04dda695f044ebad1f4ec1bda9d75d
I20240503 02:12:17.487363 5114 ts_tablet_manager.cc:1875] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.487521 5113 raft_consensus.cc:2229] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.487761 5113 raft_consensus.cc:2258] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.488032 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.488250 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_610784c3516842aca14cc78e69be69dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.488721 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.490618 5113 ts_tablet_manager.cc:1875] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:17.488721 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_610784c3516842aca14cc78e69be69dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.499214 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1bd7febf6c46410199feb217351e80e7
I20240503 02:12:17.499909 4271 catalog_manager.cc:6554] ReplaceTablet: tablet 1bd7febf6c46410199feb217351e80e7 deleted and replaced by tablet 48135062d92949dc8386c43cf103ced4
I20240503 02:12:17.499943 5114 ts_tablet_manager.cc:1888] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.500105 5114 log.cc:1196] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/43f7e89356e541d0b404f7f8bee00a6d
I20240503 02:12:17.500129 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7febf6c46410199feb217351e80e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 48135062d92949dc8386c43cf103ced4) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.500566 5114 ts_tablet_manager.cc:1909] T 43f7e89356e541d0b404f7f8bee00a6d P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:17.501623 5114 tablet_replica.cc:330] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.501816 5114 raft_consensus.cc:2229] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:17.502157 5114 raft_consensus.cc:2258] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.503472 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
I20240503 02:12:17.504204 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7febf6c46410199feb217351e80e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 48135062d92949dc8386c43cf103ced4) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.504540 5115 tablet_replica.cc:330] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.504745 5115 raft_consensus.cc:2229] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.504999 5115 raft_consensus.cc:2258] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.505923 5115 ts_tablet_manager.cc:1875] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.511895 5115 ts_tablet_manager.cc:1888] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.512079 5115 log.cc:1196] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/1bd7febf6c46410199feb217351e80e7
I20240503 02:12:17.512565 5115 ts_tablet_manager.cc:1909] T 1bd7febf6c46410199feb217351e80e7 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:17.512625 4259 catalog_manager.cc:4909] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): alter failed for tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]): Illegal state: Tablet not RUNNING: STOPPING
I20240503 02:12:17.513769 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
I20240503 02:12:17.504364 5113 ts_tablet_manager.cc:1888] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.514251 5113 log.cc:1196] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.514763 5113 ts_tablet_manager.cc:1909] T debf4d74181d4e97a7f46dd882d0b007 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.515874 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet debf4d74181d4e97a7f46dd882d0b007 (table default.Table_c24535208ac44ab8925d8bb9f586a326 [id=0bf2d5d9d1e841979be503abc86e5763]) successfully deleted
I20240503 02:12:17.516247 5113 tablet_replica.cc:330] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.509532 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7febf6c46410199feb217351e80e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 48135062d92949dc8386c43cf103ced4) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.516481 5113 raft_consensus.cc:2229] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:17.505940 4746 leader_election.cc:341] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.517127 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 11ae56645b97426b9458d9b1c91206c2
I20240503 02:12:17.517660 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 11ae56645b97426b9458d9b1c91206c2 deleted and replaced by tablet c46ea1084304449e895be230d86471a9
I20240503 02:12:17.518785 4273 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_973e6bc717c34997b7ab76b30f3d64e4 [id=db26d21e021444d795aade9f022af228]). Expected version 2 got 0
I20240503 02:12:17.520169 4270 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_973e6bc717c34997b7ab76b30f3d64e4 [id=db26d21e021444d795aade9f022af228]). Expected version 2 got 0
W20240503 02:12:17.521116 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c46ea1084304449e895be230d86471a9 has no consensus state
I20240503 02:12:17.521621 5114 ts_tablet_manager.cc:1875] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.528154 5114 ts_tablet_manager.cc:1888] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.528342 5114 log.cc:1196] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/1bd7febf6c46410199feb217351e80e7
I20240503 02:12:17.528368 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.528824 5114 ts_tablet_manager.cc:1909] T 1bd7febf6c46410199feb217351e80e7 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:17.528892 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.529209 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ae56645b97426b9458d9b1c91206c2 with delete_type TABLET_DATA_DELETED (replaced by tablet c46ea1084304449e895be230d86471a9) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.529381 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.529639 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.529841 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.530041 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 5274ab7223e8424388bf5d742e5a3455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.530182 5114 tablet_replica.cc:330] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:17.530256 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ae56645b97426b9458d9b1c91206c2 with delete_type TABLET_DATA_DELETED (replaced by tablet c46ea1084304449e895be230d86471a9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.530354 5114 raft_consensus.cc:2229] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:17.530725 4977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51106: Illegal state: RaftConsensus is not running: State = Stopping
I20240503 02:12:17.530885 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
W20240503 02:12:17.531162 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 43f7e89356e541d0b404f7f8bee00a6d because the tablet was not found. No further retry: Not found: Tablet not found: 43f7e89356e541d0b404f7f8bee00a6d
I20240503 02:12:17.530700 5114 raft_consensus.cc:2258] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.532761 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet debf4d74181d4e97a7f46dd882d0b007 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.533035 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b12e0c86f214f19b0bb11721cd83f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.533254 4653 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f7e89356e541d0b404f7f8bee00a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 461a6b30ab59485ea1a2ef8ae9815e04) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.533308 4506 tablet_service.cc:1467] Processing CreateTablet for tablet b7ddc725b90349e38301cfc29c21d848 (DEFAULT_TABLE table=default.Table_97eb124b7de24c8893a1c70ec2897e7b [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.533617 5114 ts_tablet_manager.cc:1875] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.533624 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf (DEFAULT_TABLE table=default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.533787 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ddc725b90349e38301cfc29c21d848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.534003 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f54d953fb24a3ea4bb21b6cf395edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.534020 4509 tablet_service.cc:1467] Processing CreateTablet for tablet fed654900a11471eaf6f7a198f8a985b (DEFAULT_TABLE table=default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.534353 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed654900a11471eaf6f7a198f8a985b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.535873 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b7ddc725b90349e38301cfc29c21d848 (DEFAULT_TABLE table=default.Table_97eb124b7de24c8893a1c70ec2897e7b [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.536321 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ddc725b90349e38301cfc29c21d848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.537840 4505 tablet_service.cc:1467] Processing CreateTablet for tablet a2dfd4c7a6f3483cace427cd87003535 (DEFAULT_TABLE table=default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.538189 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dfd4c7a6f3483cace427cd87003535. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:17.519133 4746 leader_election.cc:341] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.538343 4746 leader_election.cc:302] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:17.538691 4746 raft_consensus.cc:2670] T 5274ab7223e8424388bf5d742e5a3455 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:17.541239 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf (DEFAULT_TABLE table=default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.541591 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f54d953fb24a3ea4bb21b6cf395edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.542819 4791 tablet_service.cc:1467] Processing CreateTablet for tablet b7ddc725b90349e38301cfc29c21d848 (DEFAULT_TABLE table=default.Table_97eb124b7de24c8893a1c70ec2897e7b [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.543354 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ddc725b90349e38301cfc29c21d848. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:17.531376 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet debf4d74181d4e97a7f46dd882d0b007 because the tablet was not found. No further retry: Not found: Tablet not found: debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.544975 4507 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee5bceac941cd83471e5cad5dd463 (DEFAULT_TABLE table=default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.545243 4790 tablet_service.cc:1467] Processing CreateTablet for tablet a2dfd4c7a6f3483cace427cd87003535 (DEFAULT_TABLE table=default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.545362 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee5bceac941cd83471e5cad5dd463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.545301 4786 tablet_service.cc:1467] Processing CreateTablet for tablet f22a4b8015354db493652d8f336a1265 (DEFAULT_TABLE table=default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.545720 4793 tablet_service.cc:1467] Processing CreateTablet for tablet fed654900a11471eaf6f7a198f8a985b (DEFAULT_TABLE table=default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.545810 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf (DEFAULT_TABLE table=default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.546110 4789 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee5bceac941cd83471e5cad5dd463 (DEFAULT_TABLE table=default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.546365 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 25406a6f7c104671ac8b11de0c93b161 (DEFAULT_TABLE table=default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.546736 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dfd4c7a6f3483cace427cd87003535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.549360 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25406a6f7c104671ac8b11de0c93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.549341 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 25406a6f7c104671ac8b11de0c93b161 (DEFAULT_TABLE table=default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.549958 5103 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:17.549909 4503 tablet_service.cc:1467] Processing CreateTablet for tablet e40cb35a8d544e85ae13377724ed4727 (DEFAULT_TABLE table=default.Table_1564b67cd15c49349918700f41524b71 [id=ecdb526541774f2cbecc22c8efc4c3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.550436 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40cb35a8d544e85ae13377724ed4727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.551782 5103 tablet_bootstrap.cc:654] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.553172 4787 tablet_service.cc:1467] Processing CreateTablet for tablet e40cb35a8d544e85ae13377724ed4727 (DEFAULT_TABLE table=default.Table_1564b67cd15c49349918700f41524b71 [id=ecdb526541774f2cbecc22c8efc4c3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.553393 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a4b8015354db493652d8f336a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.553774 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40cb35a8d544e85ae13377724ed4727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.554028 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee5bceac941cd83471e5cad5dd463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.554189 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25406a6f7c104671ac8b11de0c93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.557440 4502 tablet_service.cc:1467] Processing CreateTablet for tablet f22a4b8015354db493652d8f336a1265 (DEFAULT_TABLE table=default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.557884 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac (DEFAULT_TABLE table=default.Table_db3a4a31a6ad4d81914cba26c1890d66 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.558040 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a4b8015354db493652d8f336a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.558396 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 634beeb9d8394b1da8c4ad79ba7e26ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.558737 4273 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_973e6bc717c34997b7ab76b30f3d64e4 [id=db26d21e021444d795aade9f022af228]). Expected version 2 got 0
I20240503 02:12:17.561758 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 (DEFAULT_TABLE table=default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.562351 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6b30ab59485ea1a2ef8ae9815e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.565634 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f54d953fb24a3ea4bb21b6cf395edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.569074 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed654900a11471eaf6f7a198f8a985b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.573156 5105 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.574882 5105 tablet_bootstrap.cc:654] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.575902 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 48135062d92949dc8386c43cf103ced4 (DEFAULT_TABLE table=default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.576376 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48135062d92949dc8386c43cf103ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.577100 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 (DEFAULT_TABLE table=default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.577697 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6b30ab59485ea1a2ef8ae9815e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.516701 5113 raft_consensus.cc:2258] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.579422 5113 ts_tablet_manager.cc:1875] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.586875 4784 tablet_service.cc:1467] Processing CreateTablet for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (DEFAULT_TABLE table=default.Table_610784c3516842aca14cc78e69be69dc [id=4d9a163c0c564955a47a65f799a0018c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.587213 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac (DEFAULT_TABLE table=default.Table_db3a4a31a6ad4d81914cba26c1890d66 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.587700 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 634beeb9d8394b1da8c4ad79ba7e26ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.588083 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c46ea1084304449e895be230d86471a9 (DEFAULT_TABLE table=default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.588724 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ea1084304449e895be230d86471a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.591008 5113 ts_tablet_manager.cc:1888] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.591169 5113 log.cc:1196] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/43f7e89356e541d0b404f7f8bee00a6d
W20240503 02:12:17.591466 4259 catalog_manager.cc:4903] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): alter failed for tablet 11ae56645b97426b9458d9b1c91206c2 (table default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:17.591668 5113 ts_tablet_manager.cc:1909] T 43f7e89356e541d0b404f7f8bee00a6d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.592235 4508 tablet_service.cc:1467] Processing CreateTablet for tablet c46ea1084304449e895be230d86471a9 (DEFAULT_TABLE table=default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.592682 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0d62ff4ac2460a9620a8e16e5d6548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.592718 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ea1084304449e895be230d86471a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.594431 4501 tablet_service.cc:1467] Processing CreateTablet for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (DEFAULT_TABLE table=default.Table_610784c3516842aca14cc78e69be69dc [id=4d9a163c0c564955a47a65f799a0018c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.594827 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0d62ff4ac2460a9620a8e16e5d6548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.594841 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 43f7e89356e541d0b404f7f8bee00a6d (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
I20240503 02:12:17.595314 5113 tablet_replica.cc:330] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.596333 5113 raft_consensus.cc:2229] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.596557 5113 raft_consensus.cc:2258] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.598156 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_73fd503758ae407593561d3ad8412920" } modify_external_catalogs: true
I20240503 02:12:17.598461 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_73fd503758ae407593561d3ad8412920" } modify_external_catalogs: true
I20240503 02:12:17.600435 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_06755c09ee504c3d8469e8b4d2acf8a3" } modify_external_catalogs: true
I20240503 02:12:17.600672 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_06755c09ee504c3d8469e8b4d2acf8a3" } modify_external_catalogs: true
I20240503 02:12:17.602300 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 48135062d92949dc8386c43cf103ced4 (DEFAULT_TABLE table=default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.602766 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48135062d92949dc8386c43cf103ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.609458 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_94169f163e904089be031c1def8b5ba6" } modify_external_catalogs: true
I20240503 02:12:17.609767 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_94169f163e904089be031c1def8b5ba6" } modify_external_catalogs: true
I20240503 02:12:17.609870 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_610784c3516842aca14cc78e69be69dc" } new_table_name: "default.Table_8513e57530cf4b839dbe5a8a0995ac31" modify_external_catalogs: true
I20240503 02:12:17.611351 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_db3a4a31a6ad4d81914cba26c1890d66" } new_table_name: "default.Table_9b239c7d723447fca17caaddebdc65e9" modify_external_catalogs: true
W20240503 02:12:17.611914 4259 catalog_manager.cc:4903] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): alter failed for tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]),no further retry: Not found: Tablet not found: 1bd7febf6c46410199feb217351e80e7
I20240503 02:12:17.612581 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_91dc254b83f74fcaa5e18ac066681ce6" } new_table_name: "default.Table_ee086c1536f043f88f2aa8d78c795ce3" modify_external_catalogs: true
I20240503 02:12:17.612751 5113 ts_tablet_manager.cc:1875] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.613116 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1564b67cd15c49349918700f41524b71" } new_table_name: "default.Table_e7595a16e3644580916a173219adfc6b" modify_external_catalogs: true
I20240503 02:12:17.614495 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908" } modify_external_catalogs: true
I20240503 02:12:17.614710 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908" } modify_external_catalogs: true
I20240503 02:12:17.609524 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_aecae221afdc4f5ebe0462800c22c19f" } modify_external_catalogs: true
I20240503 02:12:17.615849 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_aecae221afdc4f5ebe0462800c22c19f" } modify_external_catalogs: true
I20240503 02:12:17.626044 5105 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.626268 5105 ts_tablet_manager.cc:1367] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.010s
I20240503 02:12:17.626471 5113 ts_tablet_manager.cc:1888] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.626648 5113 log.cc:1196] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1bd7febf6c46410199feb217351e80e7
I20240503 02:12:17.627064 5113 ts_tablet_manager.cc:1909] T 1bd7febf6c46410199feb217351e80e7 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.627194 5105 raft_consensus.cc:348] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.627424 5105 raft_consensus.cc:374] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.627530 5105 raft_consensus.cc:729] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.627796 5105 consensus_queue.cc:260] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.628199 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1bd7febf6c46410199feb217351e80e7 (table default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
I20240503 02:12:17.628362 5105 ts_tablet_manager.cc:1398] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:17.628648 5105 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.629328 5103 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:17.629490 5103 ts_tablet_manager.cc:1367] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.002s
I20240503 02:12:17.630254 5103 raft_consensus.cc:348] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.630976 5103 raft_consensus.cc:374] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.631078 5103 raft_consensus.cc:729] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.631285 5126 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:17.631351 5103 consensus_queue.cc:260] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.635555 5103 ts_tablet_manager.cc:1398] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:17.636667 5105 tablet_bootstrap.cc:654] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.638062 5103 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:17.639787 5103 tablet_bootstrap.cc:654] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.641000 5106 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:17.641014 5115 tablet_replica.cc:330] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:17.641741 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6b99fe59a9ab4f7ea06c64c24c3f45b3 because the tablet was not found. No further retry: Not found: Tablet not found: 6b99fe59a9ab4f7ea06c64c24c3f45b3
W20240503 02:12:17.642047 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1b12e0c86f214f19b0bb11721cd83f29 because the tablet was not found. No further retry: Not found: Tablet not found: 1b12e0c86f214f19b0bb11721cd83f29
W20240503 02:12:17.642514 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet debf4d74181d4e97a7f46dd882d0b007 because the tablet was not found. No further retry: Not found: Tablet not found: debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.644752 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ae56645b97426b9458d9b1c91206c2 with delete_type TABLET_DATA_DELETED (replaced by tablet c46ea1084304449e895be230d86471a9) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:17.645340 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5274ab7223e8424388bf5d742e5a3455 because the tablet was not found. No further retry: Not found: Tablet not found: 5274ab7223e8424388bf5d742e5a3455
I20240503 02:12:17.645409 5113 tablet_replica.cc:330] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:17.645632 5113 raft_consensus.cc:2229] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.645568 4651 tablet_service.cc:1467] Processing CreateTablet for tablet f22a4b8015354db493652d8f336a1265 (DEFAULT_TABLE table=default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.645839 5113 raft_consensus.cc:2258] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.645779 4655 tablet_service.cc:1467] Processing CreateTablet for tablet e40cb35a8d544e85ae13377724ed4727 (DEFAULT_TABLE table=default.Table_1564b67cd15c49349918700f41524b71 [id=ecdb526541774f2cbecc22c8efc4c3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.644320 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 25406a6f7c104671ac8b11de0c93b161 (DEFAULT_TABLE table=default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.647233 5113 ts_tablet_manager.cc:1875] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.648425 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a2dfd4c7a6f3483cace427cd87003535
I20240503 02:12:17.648509 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:17.650367 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dfd4c7a6f3483cace427cd87003535 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.650709 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dfd4c7a6f3483cace427cd87003535 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.651168 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.656572 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.664899 5113 ts_tablet_manager.cc:1888] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.665086 5113 log.cc:1196] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/11ae56645b97426b9458d9b1c91206c2
I20240503 02:12:17.665589 5113 ts_tablet_manager.cc:1909] T 11ae56645b97426b9458d9b1c91206c2 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:17.669973 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9e283dbc214c46a28983ff2cb567ee77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.670578 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e283dbc214c46a28983ff2cb567ee77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.673063 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 11ae56645b97426b9458d9b1c91206c2 (table default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]) successfully deleted
I20240503 02:12:17.673251 5103 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
W20240503 02:12:17.673367 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a2dfd4c7a6f3483cace427cd87003535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dfd4c7a6f3483cace427cd87003535 already in progress: creating tablet
I20240503 02:12:17.673455 5103 ts_tablet_manager.cc:1367] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.009s
W20240503 02:12:17.673666 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet e0fee5bceac941cd83471e5cad5dd463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fee5bceac941cd83471e5cad5dd463 already in progress: creating tablet
I20240503 02:12:17.674281 5103 raft_consensus.cc:348] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.674546 5103 raft_consensus.cc:374] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.674634 5103 raft_consensus.cc:729] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.674943 5103 consensus_queue.cc:260] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:17.675781 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 634beeb9d8394b1da8c4ad79ba7e26ac consensus state has no leader
I20240503 02:12:17.675948 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_337d15b629af4cd3be74fc053b72fbef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.676308 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_71f9f712282348f1b44f2b37d9962c96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.676548 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_337d15b629af4cd3be74fc053b72fbef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:17.676997 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71f9f712282348f1b44f2b37d9962c96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.679073 5103 ts_tablet_manager.cc:1398] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240503 02:12:17.679119 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e40cb35a8d544e85ae13377724ed4727 consensus state has no leader
I20240503 02:12:17.679716 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bfb80f32280f4ccc9db82947883b0956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.680383 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfb80f32280f4ccc9db82947883b0956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.681291 5103 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:17.682885 5103 tablet_bootstrap.cc:654] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.685966 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 88f54d953fb24a3ea4bb21b6cf395edf
W20240503 02:12:17.687186 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd0d62ff4ac2460a9620a8e16e5d6548 consensus state has no leader
I20240503 02:12:17.687731 4268 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 634beeb9d8394b1da8c4ad79ba7e26ac (table default.Table_9b239c7d723447fca17caaddebdc65e9 [id=536a3d22f6e34248b7936ba2e7a05fd5]). Expected version 1 got 0
I20240503 02:12:17.688135 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c46ea1084304449e895be230d86471a9
I20240503 02:12:17.687747 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6c3c676734a64db68d02daa66f4d8ee1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.693667 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c3c676734a64db68d02daa66f4d8ee1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:17.699165 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48135062d92949dc8386c43cf103ced4 consensus state has no leader
I20240503 02:12:17.699827 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f22a4b8015354db493652d8f336a1265
I20240503 02:12:17.702401 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0fc326172fb94ff3957ddea1033b4a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.703153 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fc326172fb94ff3957ddea1033b4a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.707048 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_af0acc1c80404f3a875b88f459374f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.707906 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af0acc1c80404f3a875b88f459374f81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.710676 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3ae28439fd6b46438f67fd812635c2f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.714871 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ae28439fd6b46438f67fd812635c2f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.717442 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.718183 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.718487 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.688829 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7b1ece612f9744a79e3c49f09e7ddb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:17.718751 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:17.719182 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b1ece612f9744a79e3c49f09e7ddb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.719903 5105 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.720111 5105 ts_tablet_manager.cc:1367] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.091s user 0.033s sys 0.034s
W20240503 02:12:17.720888 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 88f54d953fb24a3ea4bb21b6cf395edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88f54d953fb24a3ea4bb21b6cf395edf already in progress: creating tablet
I20240503 02:12:17.720976 5105 raft_consensus.cc:348] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:17.721228 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f22a4b8015354db493652d8f336a1265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f22a4b8015354db493652d8f336a1265 already in progress: creating tablet
I20240503 02:12:17.721220 5105 raft_consensus.cc:374] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.721136 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:17.721385 5105 raft_consensus.cc:729] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.721716 5105 consensus_queue.cc:260] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:17.722827 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c46ea1084304449e895be230d86471a9 because the tablet was not found. No further retry: Not found: Tablet not found: c46ea1084304449e895be230d86471a9
I20240503 02:12:17.725977 5105 ts_tablet_manager.cc:1398] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:17.727661 5106 tablet_bootstrap.cc:654] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.727970 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:17.730361 5105 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.730358 5114 ts_tablet_manager.cc:1888] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:17.730500 5114 log.cc:1196] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/11ae56645b97426b9458d9b1c91206c2
I20240503 02:12:17.730652 4654 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee5bceac941cd83471e5cad5dd463 (DEFAULT_TABLE table=default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.732625 5105 tablet_bootstrap.cc:654] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.732787 4648 tablet_service.cc:1467] Processing CreateTablet for tablet fed654900a11471eaf6f7a198f8a985b (DEFAULT_TABLE table=default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.733839 5115 raft_consensus.cc:2229] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.734902 4652 tablet_service.cc:1467] Processing CreateTablet for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (DEFAULT_TABLE table=default.Table_610784c3516842aca14cc78e69be69dc [id=4d9a163c0c564955a47a65f799a0018c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.735275 4649 tablet_service.cc:1467] Processing CreateTablet for tablet a2dfd4c7a6f3483cace427cd87003535 (DEFAULT_TABLE table=default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.736579 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22a4b8015354db493652d8f336a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.737171 5105 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.737325 5105 ts_tablet_manager.cc:1367] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:12:17.738183 5105 raft_consensus.cc:348] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.738417 5105 raft_consensus.cc:374] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.738391 5115 raft_consensus.cc:2258] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.738519 5105 raft_consensus.cc:729] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.738804 5105 consensus_queue.cc:260] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.739430 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0d62ff4ac2460a9620a8e16e5d6548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.745250 4644 tablet_service.cc:1467] Processing CreateTablet for tablet c46ea1084304449e895be230d86471a9 (DEFAULT_TABLE table=default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.746413 5105 ts_tablet_manager.cc:1398] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:17.747148 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac (DEFAULT_TABLE table=default.Table_db3a4a31a6ad4d81914cba26c1890d66 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.749114 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25406a6f7c104671ac8b11de0c93b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.750361 5114 ts_tablet_manager.cc:1909] T 11ae56645b97426b9458d9b1c91206c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:17.754089 5105 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.755973 5105 tablet_bootstrap.cc:654] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.762933 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dfd4c7a6f3483cace427cd87003535 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.763464 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 (DEFAULT_TABLE table=default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.764163 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40cb35a8d544e85ae13377724ed4727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.764525 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dfd4c7a6f3483cace427cd87003535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.764690 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.765136 5115 ts_tablet_manager.cc:1875] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:17.773409 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed654900a11471eaf6f7a198f8a985b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.774384 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6b30ab59485ea1a2ef8ae9815e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.774699 4970 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:17.774892 4970 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.775627 4970 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:17.776561 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:17.777168 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 634beeb9d8394b1da8c4ad79ba7e26ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:17.777341 4746 leader_election.cc:341] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: e40cb35a8d544e85ae13377724ed4727
I20240503 02:12:17.781180 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ea1084304449e895be230d86471a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:17.781635 4746 leader_election.cc:341] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:17.781899 4746 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:17.782686 4970 raft_consensus.cc:2726] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:17.773442 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.785382 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee5bceac941cd83471e5cad5dd463. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:17.809319 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 69885.9us
I20240503 02:12:17.814051 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 634beeb9d8394b1da8c4ad79ba7e26ac (table default.Table_9b239c7d723447fca17caaddebdc65e9 [id=536a3d22f6e34248b7936ba2e7a05fd5]). Expected version 1 got 0
I20240503 02:12:17.818733 5105 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.818984 5105 ts_tablet_manager.cc:1367] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.015s
I20240503 02:12:17.820082 5105 raft_consensus.cc:348] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.820430 5105 raft_consensus.cc:374] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.820524 5105 raft_consensus.cc:729] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.820868 5105 consensus_queue.cc:260] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.823483 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.826228 5105 ts_tablet_manager.cc:1398] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:17.831044 5105 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.833508 5105 tablet_bootstrap.cc:654] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.821998 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 48135062d92949dc8386c43cf103ced4 (DEFAULT_TABLE table=default.Table_91dc254b83f74fcaa5e18ac066681ce6 [id=8b3bd9d6487e473089474aa5cc736cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:17.839161 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48135062d92949dc8386c43cf103ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.845891 5105 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.846138 5105 ts_tablet_manager.cc:1367] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240503 02:12:17.846975 5105 raft_consensus.cc:348] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.847221 5105 raft_consensus.cc:374] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.847312 5105 raft_consensus.cc:729] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.847609 5105 consensus_queue.cc:260] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:17.849126 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a2dfd4c7a6f3483cace427cd87003535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dfd4c7a6f3483cace427cd87003535 already in progress: creating tablet
I20240503 02:12:17.853142 5105 ts_tablet_manager.cc:1398] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240503 02:12:17.860610 5105 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:17.862895 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet e0fee5bceac941cd83471e5cad5dd463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fee5bceac941cd83471e5cad5dd463 already in progress: creating tablet
W20240503 02:12:17.863404 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 88f54d953fb24a3ea4bb21b6cf395edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88f54d953fb24a3ea4bb21b6cf395edf already in progress: creating tablet
W20240503 02:12:17.863826 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c46ea1084304449e895be230d86471a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c46ea1084304449e895be230d86471a9 already in progress: creating tablet
W20240503 02:12:17.864154 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f22a4b8015354db493652d8f336a1265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f22a4b8015354db493652d8f336a1265 already in progress: creating tablet
W20240503 02:12:17.864632 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet debf4d74181d4e97a7f46dd882d0b007 because the tablet was not found. No further retry: Not found: Tablet not found: debf4d74181d4e97a7f46dd882d0b007
I20240503 02:12:17.866182 5105 tablet_bootstrap.cc:654] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.872895 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:17.848310 5125 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:17.876344 5105 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:17.876611 5105 ts_tablet_manager.cc:1367] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240503 02:12:17.877705 5105 raft_consensus.cc:348] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.878053 5105 raft_consensus.cc:374] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.878144 5105 raft_consensus.cc:729] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.878535 5105 consensus_queue.cc:260] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.888290 5105 ts_tablet_manager.cc:1398] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:12:17.850193 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f396f63d98b5441d8f9ccc43949c520e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:17.892386 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f396f63d98b5441d8f9ccc43949c520e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:17.918467 5105 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:17.920199 5106 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:17.920532 5106 ts_tablet_manager.cc:1367] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.321s user 0.006s sys 0.001s
I20240503 02:12:17.921067 5105 tablet_bootstrap.cc:654] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.923775 5106 raft_consensus.cc:348] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.924265 5106 raft_consensus.cc:374] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:17.924614 5106 raft_consensus.cc:729] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:17.925102 5106 consensus_queue.cc:260] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.933293 5115 ts_tablet_manager.cc:1888] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:17.934188 5106 ts_tablet_manager.cc:1398] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:17.945667 5115 log.cc:1196] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/1b12e0c86f214f19b0bb11721cd83f29
I20240503 02:12:17.946396 5115 ts_tablet_manager.cc:1909] T 1b12e0c86f214f19b0bb11721cd83f29 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:17.948434 5115 tablet_replica.cc:330] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:17.948849 5115 raft_consensus.cc:2229] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.949265 5115 raft_consensus.cc:2258] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.950170 5135 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:17.951948 4970 raft_consensus.cc:482] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:17.952160 4970 raft_consensus.cc:504] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.952852 4970 leader_election.cc:288] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:17.955027 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ddc725b90349e38301cfc29c21d848" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:17.955334 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 1b12e0c86f214f19b0bb11721cd83f29 (table default.Table_4ba3396bd3d040bebb3c9099309e615c [id=bf3f22f82f994ea3a96b7dc2c8569fe6]) successfully deleted
W20240503 02:12:17.955698 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 43f7e89356e541d0b404f7f8bee00a6d because the tablet was not found. No further retry: Not found: Tablet not found: 43f7e89356e541d0b404f7f8bee00a6d
I20240503 02:12:17.959671 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ddc725b90349e38301cfc29c21d848" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:17.959995 4529 raft_consensus.cc:2454] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:17.960564 4746 leader_election.cc:302] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:17.960942 5106 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:17.960958 4970 raft_consensus.cc:2781] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:17.961083 4970 raft_consensus.cc:482] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:17.961220 4970 raft_consensus.cc:3037] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:17.964416 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ee086c1536f043f88f2aa8d78c795ce3" } new_table_name: "default.Table_c623ef45b41e425fb3d1457c5cffb85d" modify_external_catalogs: true
I20240503 02:12:17.968497 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 11ae56645b97426b9458d9b1c91206c2 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.969197 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 88f54d953fb24a3ea4bb21b6cf395edf from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.969313 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b12e0c86f214f19b0bb11721cd83f29 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.969486 4970 raft_consensus.cc:504] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.969766 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b12e0c86f214f19b0bb11721cd83f29 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.969808 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 88f54d953fb24a3ea4bb21b6cf395edf from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:17.970376 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c0b1f5e771194ca79a9b98901d09ee64" } modify_external_catalogs: true
I20240503 02:12:17.970491 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9e283dbc214c46a28983ff2cb567ee77" } modify_external_catalogs: true
I20240503 02:12:17.970580 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c0b1f5e771194ca79a9b98901d09ee64" } modify_external_catalogs: true
I20240503 02:12:17.970710 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9e283dbc214c46a28983ff2cb567ee77" } modify_external_catalogs: true
I20240503 02:12:17.970883 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ddc725b90349e38301cfc29c21d848" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:17.971097 4528 raft_consensus.cc:3037] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:17.973446 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_973e6bc717c34997b7ab76b30f3d64e4" } new_table_name: "default.Table_85eb2a2a9c2b42d4bad6242f8e83006c" modify_external_catalogs: true
I20240503 02:12:17.974072 4528 raft_consensus.cc:2454] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:17.974638 4746 leader_election.cc:302] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:17.974639 4270 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_973e6bc717c34997b7ab76b30f3d64e4 [id=db26d21e021444d795aade9f022af228] alter complete (version 2)
I20240503 02:12:17.975234 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8513e57530cf4b839dbe5a8a0995ac31" } new_table_name: "default.Table_ad0ceaed9ff641a8815b73da0ae0e02c" modify_external_catalogs: true
I20240503 02:12:17.975919 4970 leader_election.cc:288] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:17.976120 4970 raft_consensus.cc:2781] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:17.976274 4970 raft_consensus.cc:686] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:17.976505 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e37722f22364e62b5e10db841f1b963" } modify_external_catalogs: true
I20240503 02:12:17.976706 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e37722f22364e62b5e10db841f1b963" } modify_external_catalogs: true
I20240503 02:12:17.978530 4970 consensus_queue.cc:237] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:17.982517 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_75654fad5307429586e50445bc45687f" } modify_external_catalogs: true
I20240503 02:12:17.982821 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_75654fad5307429586e50445bc45687f" } modify_external_catalogs: true
I20240503 02:12:17.986979 5106 tablet_bootstrap.cc:654] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:17.998086 5115 ts_tablet_manager.cc:1875] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:17.999265 4746 leader_election.cc:341] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.002264 5106 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.003476 5106 ts_tablet_manager.cc:1367] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.001s
I20240503 02:12:18.003873 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ddc725b90349e38301cfc29c21d848" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:18.005028 4746 leader_election.cc:341] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.006810 5106 raft_consensus.cc:348] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.007066 5106 raft_consensus.cc:374] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.007150 5106 raft_consensus.cc:729] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.007509 5106 consensus_queue.cc:260] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.014413 5106 ts_tablet_manager.cc:1398] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:12:18.020570 5106 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:18.024153 5103 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.024366 5103 ts_tablet_manager.cc:1367] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.343s user 0.142s sys 0.170s
I20240503 02:12:18.025449 5103 raft_consensus.cc:348] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.025552 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9b239c7d723447fca17caaddebdc65e9" } new_table_name: "default.Table_28877a245811426292a39f20283988a2" modify_external_catalogs: true
I20240503 02:12:18.025732 5103 raft_consensus.cc:374] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.025843 5103 raft_consensus.cc:729] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.026180 5103 consensus_queue.cc:260] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.030026 5103 ts_tablet_manager.cc:1398] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:18.032529 5103 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.034332 5103 tablet_bootstrap.cc:654] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.038535 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_e7595a16e3644580916a173219adfc6b [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 1 got 0
I20240503 02:12:18.038870 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 461a6b30ab59485ea1a2ef8ae9815e04 (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]). Expected version 2 got 0
I20240503 02:12:18.039062 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_8513e57530cf4b839dbe5a8a0995ac31 [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 1 got 0
I20240503 02:12:18.042562 5103 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.042505 5141 raft_consensus.cc:482] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.042754 5103 ts_tablet_manager.cc:1367] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:12:18.042775 5141 raft_consensus.cc:504] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.043716 5141 leader_election.cc:288] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:18.043695 5103 raft_consensus.cc:348] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.043941 5103 raft_consensus.cc:374] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.044023 5103 raft_consensus.cc:729] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.044348 5103 consensus_queue.cc:260] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.044498 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461a6b30ab59485ea1a2ef8ae9815e04" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:18.044610 4646 tablet_service.cc:1467] Processing CreateTablet for tablet effb2ce36a31477382865461d89917be (DEFAULT_TABLE table=default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.044982 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461a6b30ab59485ea1a2ef8ae9815e04" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:18.045706 4746 leader_election.cc:341] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.045917 4783 tablet_service.cc:1467] Processing CreateTablet for tablet effb2ce36a31477382865461d89917be (DEFAULT_TABLE table=default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.046088 4746 leader_election.cc:341] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.046108 4506 tablet_service.cc:1467] Processing CreateTablet for tablet effb2ce36a31477382865461d89917be (DEFAULT_TABLE table=default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.046208 4746 leader_election.cc:302] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:18.046384 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effb2ce36a31477382865461d89917be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.046454 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effb2ce36a31477382865461d89917be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.046615 5141 raft_consensus.cc:2726] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:18.047569 5103 ts_tablet_manager.cc:1398] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:18.048790 5106 tablet_bootstrap.cc:654] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.050266 5103 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.067916 5103 tablet_bootstrap.cc:654] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.080060 5103 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.080297 5103 ts_tablet_manager.cc:1367] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20240503 02:12:18.081480 5103 raft_consensus.cc:348] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.081820 5103 raft_consensus.cc:374] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.081939 5103 raft_consensus.cc:729] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.082229 5103 consensus_queue.cc:260] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.082914 5103 ts_tablet_manager.cc:1398] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:18.083352 5103 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.085654 5103 tablet_bootstrap.cc:654] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.092799 5103 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.093039 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effb2ce36a31477382865461d89917be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.093413 5103 ts_tablet_manager.cc:1367] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:12:18.094242 5103 raft_consensus.cc:348] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.094492 5103 raft_consensus.cc:374] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.094558 5103 raft_consensus.cc:729] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.094805 5103 consensus_queue.cc:260] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.097538 5103 ts_tablet_manager.cc:1398] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:18.098085 5103 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.100677 5103 tablet_bootstrap.cc:654] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.110651 5103 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.110827 5103 ts_tablet_manager.cc:1367] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240503 02:12:18.111533 5105 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.111577 5103 raft_consensus.cc:348] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.111801 5103 raft_consensus.cc:374] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.111877 5105 ts_tablet_manager.cc:1367] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.193s user 0.044s sys 0.064s
I20240503 02:12:18.111884 5103 raft_consensus.cc:729] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.112679 5105 raft_consensus.cc:348] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.113045 5105 raft_consensus.cc:374] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.113183 5105 raft_consensus.cc:729] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.113327 5103 consensus_queue.cc:260] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.113482 5105 consensus_queue.cc:260] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.116977 5103 ts_tablet_manager.cc:1398] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240503 02:12:18.117308 5103 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.119117 5103 tablet_bootstrap.cc:654] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:18.122318 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48135062d92949dc8386c43cf103ced4 consensus state has no leader
I20240503 02:12:18.122642 5105 ts_tablet_manager.cc:1398] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:12:18.127285 5105 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:18.129191 5105 tablet_bootstrap.cc:654] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.145215 4273 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_8513e57530cf4b839dbe5a8a0995ac31 [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 1 got 0
W20240503 02:12:18.154654 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd0d62ff4ac2460a9620a8e16e5d6548 consensus state has no leader
I20240503 02:12:18.163532 5106 tablet_bootstrap.cc:492] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.163659 5141 raft_consensus.cc:482] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.163787 5106 ts_tablet_manager.cc:1367] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.143s user 0.033s sys 0.036s
I20240503 02:12:18.163849 5141 raft_consensus.cc:504] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.164594 5141 leader_election.cc:288] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:18.164777 5106 raft_consensus.cc:348] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.165053 5106 raft_consensus.cc:374] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.165149 5106 raft_consensus.cc:729] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.165359 5106 consensus_queue.cc:260] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.165444 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.165566 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:18.165654 4674 raft_consensus.cc:2454] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:18.165930 5106 ts_tablet_manager.cc:1398] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:18.166208 5106 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:18.166774 4746 leader_election.cc:341] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.167534 4746 leader_election.cc:302] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:18.167831 5141 raft_consensus.cc:2781] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:18.167935 5106 tablet_bootstrap.cc:654] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.167955 5141 raft_consensus.cc:482] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:18.168056 5141 raft_consensus.cc:3037] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.170220 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_58a13c96da8f4ca089be3162a5b53abb" } modify_external_catalogs: true
I20240503 02:12:18.170493 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_58a13c96da8f4ca089be3162a5b53abb" } modify_external_catalogs: true
I20240503 02:12:18.172936 5141 raft_consensus.cc:504] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.173727 5141 leader_election.cc:288] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:18.178023 5115 ts_tablet_manager.cc:1888] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:18.178220 5115 log.cc:1196] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/11ae56645b97426b9458d9b1c91206c2
I20240503 02:12:18.178784 5115 ts_tablet_manager.cc:1909] T 11ae56645b97426b9458d9b1c91206c2 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:18.183665 5106 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.184736 5106 ts_tablet_manager.cc:1367] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20240503 02:12:18.185721 5106 raft_consensus.cc:348] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.186584 5106 raft_consensus.cc:374] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.186683 5106 raft_consensus.cc:729] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.186815 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:18.186982 5106 consensus_queue.cc:260] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:18.187737 4746 leader_election.cc:341] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.188704 5106 ts_tablet_manager.cc:1398] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:18.189069 5106 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:18.193689 5106 tablet_bootstrap.cc:654] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.197206 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e7595a16e3644580916a173219adfc6b" } new_table_name: "default.Table_5c37ffeee865425989dc0aa0cc41169a" modify_external_catalogs: true
I20240503 02:12:18.186983 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:18.199213 4674 raft_consensus.cc:3037] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.202831 4674 raft_consensus.cc:2454] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:18.203372 4746 leader_election.cc:302] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:18.203902 5141 raft_consensus.cc:2781] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:18.204051 5141 raft_consensus.cc:686] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:18.204384 5141 consensus_queue.cc:237] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.207515 4988 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.207700 4988 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.208424 4988 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.209241 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:18.209493 4812 raft_consensus.cc:2454] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:18.209376 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c942f05514a247888a6fde1dfaf00348"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.210176 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c942f05514a247888a6fde1dfaf00348 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.210251 4462 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:18.210294 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.210561 4988 raft_consensus.cc:2781] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:18.210670 4988 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:18.210758 4988 raft_consensus.cc:3037] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.213016 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.213310 4462 leader_election.cc:341] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:18.213743 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.214395 4988 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.216012 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:18.216207 4812 raft_consensus.cc:3037] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.219626 4812 raft_consensus.cc:2454] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:18.220271 4462 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:18.220862 4988 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.221230 4988 raft_consensus.cc:2781] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:18.222151 4988 raft_consensus.cc:686] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:18.222426 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:18.223104 4988 consensus_queue.cc:237] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.228000 5115 tablet_replica.cc:330] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:18.228261 5115 raft_consensus.cc:2229] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.228463 5115 raft_consensus.cc:2258] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.229305 5115 ts_tablet_manager.cc:1875] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.232144 5125 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 02:12:18.239075 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 634beeb9d8394b1da8c4ad79ba7e26ac consensus state has no leader
I20240503 02:12:18.241247 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a7fb095705c641ce929d9cc5029de3b9
I20240503 02:12:18.242715 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet effb2ce36a31477382865461d89917be
I20240503 02:12:18.242916 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fb095705c641ce929d9cc5029de3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.243487 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f1c2ade4126d49df8aebe97256f2844f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.244064 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c2ade4126d49df8aebe97256f2844f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.244055 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fb095705c641ce929d9cc5029de3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.246647 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.247401 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:18.247879 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a7fb095705c641ce929d9cc5029de3b9 because the tablet was not found. No further retry: Not found: Tablet not found: a7fb095705c641ce929d9cc5029de3b9
I20240503 02:12:18.248381 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fb095705c641ce929d9cc5029de3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.248778 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:18.248800 4462 leader_election.cc:341] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:18.249315 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet effb2ce36a31477382865461d89917be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet effb2ce36a31477382865461d89917be already in progress: creating tablet
I20240503 02:12:18.251348 4268 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_e7595a16e3644580916a173219adfc6b [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 1 got 0
I20240503 02:12:18.252388 4266 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_ad0ceaed9ff641a8815b73da0ae0e02c [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 2 got 0
I20240503 02:12:18.253108 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:18.253851 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet fed654900a11471eaf6f7a198f8a985b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.253918 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a38d73ea23ac44cab66f3cabba1f9fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.254483 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.254513 5114 tablet_replica.cc:330] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:18.254554 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet fed654900a11471eaf6f7a198f8a985b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.254757 5114 raft_consensus.cc:2229] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.254932 5114 raft_consensus.cc:2258] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.256328 5114 ts_tablet_manager.cc:1875] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.256425 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet fed654900a11471eaf6f7a198f8a985b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.258257 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_eee1175b622840fe9e2ffb544bcfd426"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.258839 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eee1175b622840fe9e2ffb544bcfd426 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.261080 5160 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:18.261988 5135 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:18.266630 5114 ts_tablet_manager.cc:1888] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.266831 5114 log.cc:1196] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:18.267248 5114 ts_tablet_manager.cc:1909] T fed654900a11471eaf6f7a198f8a985b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:18.268225 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:18.269052 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.269331 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet fed654900a11471eaf6f7a198f8a985b (table default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]) successfully deleted
W20240503 02:12:18.269631 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a7fb095705c641ce929d9cc5029de3b9 because the tablet was not found. No further retry: Not found: Tablet not found: a7fb095705c641ce929d9cc5029de3b9
W20240503 02:12:18.269896 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet effb2ce36a31477382865461d89917be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet effb2ce36a31477382865461d89917be already in progress: creating tablet
W20240503 02:12:18.270200 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fed654900a11471eaf6f7a198f8a985b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fed654900a11471eaf6f7a198f8a985b already in progress: creating tablet
I20240503 02:12:18.271644 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.273221 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.273587 5159 tablet_replica.cc:330] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:18.273685 4272 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 461a6b30ab59485ea1a2ef8ae9815e04 (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 461a6b30ab59485ea1a2ef8ae9815e04 consensus state has no leader
I20240503 02:12:18.273783 5159 raft_consensus.cc:2229] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.273975 5159 raft_consensus.cc:2258] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.274827 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dfd4c7a6f3483cace427cd87003535 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.275429 5159 ts_tablet_manager.cc:1875] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.276827 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:18.277125 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 461a6b30ab59485ea1a2ef8ae9815e04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461a6b30ab59485ea1a2ef8ae9815e04 already in progress: creating tablet
W20240503 02:12:18.286793 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e40cb35a8d544e85ae13377724ed4727 consensus state has no leader
I20240503 02:12:18.289659 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.290061 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet fed654900a11471eaf6f7a198f8a985b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.290323 5114 tablet_replica.cc:330] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:18.290525 5114 raft_consensus.cc:2229] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.290815 5114 raft_consensus.cc:2258] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.292281 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b741361e02de4b5aaed514fe7cd99f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.292866 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b741361e02de4b5aaed514fe7cd99f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.294958 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b8cf4144fb1d4392856c92576ed4dcb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.295293 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.295737 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dfd4c7a6f3483cace427cd87003535 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:18.295833 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8cf4144fb1d4392856c92576ed4dcb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.301865 5114 ts_tablet_manager.cc:1875] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.311633 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e6d61608aa724f8daa53ca5585ab2947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.312325 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6d61608aa724f8daa53ca5585ab2947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.313984 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 461a6b30ab59485ea1a2ef8ae9815e04 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.327913 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d9b602cd0e824ea185e468fd32acd15e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.328598 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9b602cd0e824ea185e468fd32acd15e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.333477 5114 ts_tablet_manager.cc:1888] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.333667 5114 log.cc:1196] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/f22a4b8015354db493652d8f336a1265
I20240503 02:12:18.334173 5114 ts_tablet_manager.cc:1909] T f22a4b8015354db493652d8f336a1265 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:18.335374 5114 tablet_replica.cc:330] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:18.335594 5114 raft_consensus.cc:2229] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.335584 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet f22a4b8015354db493652d8f336a1265 (table default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]) successfully deleted
I20240503 02:12:18.335844 5114 raft_consensus.cc:2258] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:18.336438 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fed654900a11471eaf6f7a198f8a985b because the tablet was not found. No further retry: Not found: Tablet not found: fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:18.337488 5114 ts_tablet_manager.cc:1875] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.340598 5159 ts_tablet_manager.cc:1888] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.345319 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 25406a6f7c104671ac8b11de0c93b161 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.346855 5106 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.347136 5159 log.cc:1196] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:18.347139 5106 ts_tablet_manager.cc:1367] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.158s user 0.039s sys 0.051s
I20240503 02:12:18.348623 5106 raft_consensus.cc:348] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.348948 5106 raft_consensus.cc:374] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.349076 5106 raft_consensus.cc:729] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.349365 5106 consensus_queue.cc:260] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.350004 5106 ts_tablet_manager.cc:1398] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:18.350381 5106 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:18.352155 5106 tablet_bootstrap.cc:654] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.356971 5103 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.357240 5103 ts_tablet_manager.cc:1367] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.240s user 0.052s sys 0.031s
I20240503 02:12:18.358422 5103 raft_consensus.cc:348] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.358774 5103 raft_consensus.cc:374] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.358875 5103 raft_consensus.cc:729] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.356678 5105 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.364185 5105 ts_tablet_manager.cc:1367] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.237s user 0.018s sys 0.026s
I20240503 02:12:18.364449 5159 ts_tablet_manager.cc:1909] T 461a6b30ab59485ea1a2ef8ae9815e04 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.365324 5105 raft_consensus.cc:348] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.365579 5105 raft_consensus.cc:374] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.365666 5105 raft_consensus.cc:729] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.365608 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 461a6b30ab59485ea1a2ef8ae9815e04 (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
I20240503 02:12:18.365911 5105 consensus_queue.cc:260] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.367048 5103 consensus_queue.cc:260] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.368067 5103 ts_tablet_manager.cc:1398] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240503 02:12:18.368495 5103 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.375332 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet b7ddc725b90349e38301cfc29c21d848 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.375456 5103 tablet_bootstrap.cc:654] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.376497 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 634beeb9d8394b1da8c4ad79ba7e26ac from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.377364 5105 ts_tablet_manager.cc:1398] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:12:18.377739 5159 tablet_replica.cc:330] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:18.377952 5159 raft_consensus.cc:2229] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.378130 5159 raft_consensus.cc:2258] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.378737 5159 ts_tablet_manager.cc:1875] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.381388 5105 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:18.382202 5115 ts_tablet_manager.cc:1888] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.382406 5115 log.cc:1196] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/a2dfd4c7a6f3483cace427cd87003535
I20240503 02:12:18.383388 5105 tablet_bootstrap.cc:654] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.384025 5115 ts_tablet_manager.cc:1909] T a2dfd4c7a6f3483cace427cd87003535 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:18.389966 5114 ts_tablet_manager.cc:1888] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.390344 5114 log.cc:1196] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:18.391234 5103 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.391417 5103 ts_tablet_manager.cc:1367] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240503 02:12:18.392360 5103 raft_consensus.cc:348] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.392601 5103 raft_consensus.cc:374] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.393055 5103 raft_consensus.cc:729] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.393437 5103 consensus_queue.cc:260] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.394032 5103 ts_tablet_manager.cc:1398] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:18.394382 5103 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.396140 5103 tablet_bootstrap.cc:654] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.396405 4506 tablet_service.cc:1467] Processing CreateTablet for tablet c126b8ef15c5432abf037e28a5eeba58 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.396844 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c126b8ef15c5432abf037e28a5eeba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.398202 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 634beeb9d8394b1da8c4ad79ba7e26ac from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.399677 4275 catalog_manager.cc:5484] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.400172 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_ad0ceaed9ff641a8815b73da0ae0e02c [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 2 got 0
I20240503 02:12:18.400441 4275 catalog_manager.cc:5484] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.401418 4646 tablet_service.cc:1467] Processing CreateTablet for tablet c126b8ef15c5432abf037e28a5eeba58 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.401575 4269 catalog_manager.cc:5484] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:18.401963 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c126b8ef15c5432abf037e28a5eeba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.402060 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_5c37ffeee865425989dc0aa0cc41169a [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 2 got 0
I20240503 02:12:18.403928 4792 tablet_service.cc:1467] Processing CreateTablet for tablet c126b8ef15c5432abf037e28a5eeba58 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.404512 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c126b8ef15c5432abf037e28a5eeba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.405351 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 110c67212ec345028024da32968869ba (DEFAULT_TABLE table=default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.405848 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110c67212ec345028024da32968869ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.409200 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 110c67212ec345028024da32968869ba (DEFAULT_TABLE table=default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.409426 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69392fa5f441049437639e143a7491 (DEFAULT_TABLE table=default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.409621 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110c67212ec345028024da32968869ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.409929 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69392fa5f441049437639e143a7491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.413460 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69392fa5f441049437639e143a7491 (DEFAULT_TABLE table=default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.414067 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69392fa5f441049437639e143a7491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.414311 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 110c67212ec345028024da32968869ba (DEFAULT_TABLE table=default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.414844 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110c67212ec345028024da32968869ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.417320 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69392fa5f441049437639e143a7491 (DEFAULT_TABLE table=default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.417788 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69392fa5f441049437639e143a7491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.419045 4272 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_85eb2a2a9c2b42d4bad6242f8e83006c [id=db26d21e021444d795aade9f022af228] alter complete (version 3)
I20240503 02:12:18.419847 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8c833315113b4fafb24834feb6fc6649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.420544 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c833315113b4fafb24834feb6fc6649 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.417351 4653 tablet_service.cc:1467] Processing CreateTablet for tablet b487131ac3a842009b3fe54578ea091d (DEFAULT_TABLE table=default.Table_bfb80f32280f4ccc9db82947883b0956 [id=a24019c2303b40a2adab179b7ea5ca1b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.423192 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_5c37ffeee865425989dc0aa0cc41169a [id=ecdb526541774f2cbecc22c8efc4c3dd], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e40cb35a8d544e85ae13377724ed4727 consensus state has no leader
I20240503 02:12:18.428292 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b487131ac3a842009b3fe54578ea091d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.430189 4508 tablet_service.cc:1467] Processing CreateTablet for tablet b487131ac3a842009b3fe54578ea091d (DEFAULT_TABLE table=default.Table_bfb80f32280f4ccc9db82947883b0956 [id=a24019c2303b40a2adab179b7ea5ca1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.430850 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b487131ac3a842009b3fe54578ea091d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.434666 4503 tablet_service.cc:1467] Processing CreateTablet for tablet bc0e75639a4f4f168f9d833b090ae76a (DEFAULT_TABLE table=default.Table_6c3c676734a64db68d02daa66f4d8ee1 [id=9947c6f15d2444ca895eea2f96df51ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.435114 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0e75639a4f4f168f9d833b090ae76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.435472 4788 tablet_service.cc:1467] Processing CreateTablet for tablet b487131ac3a842009b3fe54578ea091d (DEFAULT_TABLE table=default.Table_bfb80f32280f4ccc9db82947883b0956 [id=a24019c2303b40a2adab179b7ea5ca1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.435573 4645 tablet_service.cc:1467] Processing CreateTablet for tablet bc0e75639a4f4f168f9d833b090ae76a (DEFAULT_TABLE table=default.Table_6c3c676734a64db68d02daa66f4d8ee1 [id=9947c6f15d2444ca895eea2f96df51ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.435963 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0e75639a4f4f168f9d833b090ae76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.436021 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b487131ac3a842009b3fe54578ea091d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.436636 4787 tablet_service.cc:1467] Processing CreateTablet for tablet bc0e75639a4f4f168f9d833b090ae76a (DEFAULT_TABLE table=default.Table_6c3c676734a64db68d02daa66f4d8ee1 [id=9947c6f15d2444ca895eea2f96df51ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.437167 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0e75639a4f4f168f9d833b090ae76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.446643 4791 tablet_service.cc:1467] Processing CreateTablet for tablet fc887bc4db1c4e268c4c9f23096d704c (DEFAULT_TABLE table=default.Table_337d15b629af4cd3be74fc053b72fbef [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.447240 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc887bc4db1c4e268c4c9f23096d704c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.450769 5114 ts_tablet_manager.cc:1909] T e0fee5bceac941cd83471e5cad5dd463 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:18.451382 4651 tablet_service.cc:1467] Processing CreateTablet for tablet fc887bc4db1c4e268c4c9f23096d704c (DEFAULT_TABLE table=default.Table_337d15b629af4cd3be74fc053b72fbef [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.451730 5114 tablet_replica.cc:330] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:18.451934 5114 raft_consensus.cc:2229] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.452124 5114 raft_consensus.cc:2258] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.452320 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet e0fee5bceac941cd83471e5cad5dd463 (table default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]) successfully deleted
I20240503 02:12:18.456033 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc887bc4db1c4e268c4c9f23096d704c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.460439 5103 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.461277 5103 ts_tablet_manager.cc:1367] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.024s
I20240503 02:12:18.462931 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 86d719c58974442281c7d694e1ca485e (DEFAULT_TABLE table=default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.463553 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d719c58974442281c7d694e1ca485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.484395 4501 tablet_service.cc:1467] Processing CreateTablet for tablet fc887bc4db1c4e268c4c9f23096d704c (DEFAULT_TABLE table=default.Table_337d15b629af4cd3be74fc053b72fbef [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.484508 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 86d719c58974442281c7d694e1ca485e (DEFAULT_TABLE table=default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.484680 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 21e625e02b764ff2b7279b35dd10b08c (DEFAULT_TABLE table=default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.484949 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d719c58974442281c7d694e1ca485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.484978 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc887bc4db1c4e268c4c9f23096d704c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.485236 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e625e02b764ff2b7279b35dd10b08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.485169 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 86d719c58974442281c7d694e1ca485e (DEFAULT_TABLE table=default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.485635 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d719c58974442281c7d694e1ca485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.489341 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 21e625e02b764ff2b7279b35dd10b08c (DEFAULT_TABLE table=default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.489820 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e625e02b764ff2b7279b35dd10b08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.501909 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet a2dfd4c7a6f3483cace427cd87003535 (table default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]) successfully deleted
I20240503 02:12:18.513557 5159 ts_tablet_manager.cc:1888] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.514921 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:18.515970 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 634beeb9d8394b1da8c4ad79ba7e26ac
I20240503 02:12:18.516515 4274 catalog_manager.cc:6554] ReplaceTablet: tablet 634beeb9d8394b1da8c4ad79ba7e26ac deleted and replaced by tablet ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:18.519433 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c942f05514a247888a6fde1dfaf00348" } new_table_name: "default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b" modify_external_catalogs: true
I20240503 02:12:18.521639 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bfb80f32280f4ccc9db82947883b0956" } new_table_name: "default.Table_f1fb4997ae544428a3067a3b9ae447dc" modify_external_catalogs: true
I20240503 02:12:18.523365 5103 raft_consensus.cc:348] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.523576 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c623ef45b41e425fb3d1457c5cffb85d" } new_table_name: "default.Table_7941714137fb4383a724137797d96a25" modify_external_catalogs: true
I20240503 02:12:18.525251 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_337d15b629af4cd3be74fc053b72fbef" } new_table_name: "default.Table_00f06d9612844bcfba5ade0c41a86f77" modify_external_catalogs: true
I20240503 02:12:18.526510 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac with delete_type TABLET_DATA_DELETED (replaced by tablet ced251fecca84866ab3ff15bd225e2b7) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.527087 5115 tablet_replica.cc:330] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:18.527391 5115 raft_consensus.cc:2229] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.528338 5103 raft_consensus.cc:374] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.528501 5103 raft_consensus.cc:729] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.528576 5115 raft_consensus.cc:2258] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.528837 5103 consensus_queue.cc:260] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.529163 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac with delete_type TABLET_DATA_DELETED (replaced by tablet ced251fecca84866ab3ff15bd225e2b7) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.529513 5103 ts_tablet_manager.cc:1398] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20240503 02:12:18.529903 5103 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.531701 5103 tablet_bootstrap.cc:654] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:18.536136 4974 raft_consensus.cc:659] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20240503 02:12:18.536638 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 21e625e02b764ff2b7279b35dd10b08c (DEFAULT_TABLE table=default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.537324 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e625e02b764ff2b7279b35dd10b08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.539599 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 25406a6f7c104671ac8b11de0c93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.541477 5115 ts_tablet_manager.cc:1875] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.528800 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 25406a6f7c104671ac8b11de0c93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.529680 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac with delete_type TABLET_DATA_DELETED (replaced by tablet ced251fecca84866ab3ff15bd225e2b7) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.542631 5159 log.cc:1196] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a2dfd4c7a6f3483cace427cd87003535
I20240503 02:12:18.543143 5159 ts_tablet_manager.cc:1909] T a2dfd4c7a6f3483cace427cd87003535 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.544677 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a2dfd4c7a6f3483cace427cd87003535 (table default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]) successfully deleted
I20240503 02:12:18.545111 5159 tablet_replica.cc:330] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:18.545322 5159 raft_consensus.cc:2229] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.545552 5159 raft_consensus.cc:2258] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.546144 5159 ts_tablet_manager.cc:1875] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:18.548174 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7daa5af6c8044178b9f84aaba4cca33 has no consensus state
I20240503 02:12:18.550367 5114 ts_tablet_manager.cc:1875] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.550022 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.566174 5159 ts_tablet_manager.cc:1888] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.566370 5159 log.cc:1196] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:18.566892 5159 ts_tablet_manager.cc:1909] T e0fee5bceac941cd83471e5cad5dd463 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.567409 5106 tablet_bootstrap.cc:492] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.567646 5106 ts_tablet_manager.cc:1367] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.217s user 0.039s sys 0.082s
I20240503 02:12:18.568192 5159 tablet_replica.cc:330] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:18.568222 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet e0fee5bceac941cd83471e5cad5dd463 (table default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]) successfully deleted
I20240503 02:12:18.568476 5159 raft_consensus.cc:2229] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.568526 5106 raft_consensus.cc:348] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.568702 5159 raft_consensus.cc:2258] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.568787 5106 raft_consensus.cc:374] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.568878 5106 raft_consensus.cc:729] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:18.568823 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 25406a6f7c104671ac8b11de0c93b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25406a6f7c104671ac8b11de0c93b161 already in progress: creating tablet
I20240503 02:12:18.569157 5106 consensus_queue.cc:260] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.569703 5106 ts_tablet_manager.cc:1398] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:18.569940 5105 tablet_bootstrap.cc:492] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.570133 5105 ts_tablet_manager.cc:1367] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.192s user 0.041s sys 0.053s
I20240503 02:12:18.570223 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_71f9f712282348f1b44f2b37d9962c96" } modify_external_catalogs: true
I20240503 02:12:18.570390 5106 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:18.570422 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_71f9f712282348f1b44f2b37d9962c96" } modify_external_catalogs: true
I20240503 02:12:18.571027 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1c2ade4126d49df8aebe97256f2844f" } modify_external_catalogs: true
I20240503 02:12:18.571004 5105 raft_consensus.cc:348] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.571185 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1c2ade4126d49df8aebe97256f2844f" } modify_external_catalogs: true
I20240503 02:12:18.571830 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7b1ece612f9744a79e3c49f09e7ddb24" } modify_external_catalogs: true
I20240503 02:12:18.572005 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7b1ece612f9744a79e3c49f09e7ddb24" } modify_external_catalogs: true
I20240503 02:12:18.572592 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0fc326172fb94ff3957ddea1033b4a7a" } modify_external_catalogs: true
I20240503 02:12:18.572762 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 25406a6f7c104671ac8b11de0c93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.572784 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0fc326172fb94ff3957ddea1033b4a7a" } modify_external_catalogs: true
I20240503 02:12:18.571241 5105 raft_consensus.cc:374] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.574538 5105 raft_consensus.cc:729] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.574995 5105 consensus_queue.cc:260] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.575608 5105 ts_tablet_manager.cc:1398] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:18.577680 5106 tablet_bootstrap.cc:654] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.579752 5103 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.579907 5103 ts_tablet_manager.cc:1367] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.016s
I20240503 02:12:18.581630 4988 raft_consensus.cc:482] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.581815 4988 raft_consensus.cc:504] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.582551 4988 leader_election.cc:288] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.584056 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "634beeb9d8394b1da8c4ad79ba7e26ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.584420 5103 raft_consensus.cc:348] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.584645 5103 raft_consensus.cc:374] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.584739 5103 raft_consensus.cc:729] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:18.584993 4462 leader_election.cc:341] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.585197 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "634beeb9d8394b1da8c4ad79ba7e26ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:18.586073 5105 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:18.586315 4462 leader_election.cc:341] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.586495 4462 leader_election.cc:302] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:18.586970 4988 raft_consensus.cc:2726] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:18.587476 5103 consensus_queue.cc:260] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.588135 5105 tablet_bootstrap.cc:654] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:18.590030 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc887bc4db1c4e268c4c9f23096d704c consensus state has no leader
I20240503 02:12:18.590204 5159 ts_tablet_manager.cc:1875] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.593209 5103 ts_tablet_manager.cc:1398] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 02:12:18.596304 4273 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_c623ef45b41e425fb3d1457c5cffb85d [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 4 got 0
W20240503 02:12:18.597018 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48135062d92949dc8386c43cf103ced4 consensus state has no leader
I20240503 02:12:18.597133 5103 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.599392 5159 ts_tablet_manager.cc:1888] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.599659 5103 tablet_bootstrap.cc:654] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.601928 5159 log.cc:1196] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/634beeb9d8394b1da8c4ad79ba7e26ac
I20240503 02:12:18.602749 5159 ts_tablet_manager.cc:1909] T 634beeb9d8394b1da8c4ad79ba7e26ac P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.604681 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 546afc74e6c54f8aa5f7c684484d610b (DEFAULT_TABLE table=default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.604974 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 546afc74e6c54f8aa5f7c684484d610b (DEFAULT_TABLE table=default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.605213 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546afc74e6c54f8aa5f7c684484d610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.605408 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546afc74e6c54f8aa5f7c684484d610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.606547 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 14fd4fb88436498999b5f339052ac0b6 (DEFAULT_TABLE table=default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.607021 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fd4fb88436498999b5f339052ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.608450 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_97eb124b7de24c8893a1c70ec2897e7b" } new_table_name: "default.Table_0ccc60c229e74e9197d025a8c631a76c" modify_external_catalogs: true
I20240503 02:12:18.610872 4267 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_7941714137fb4383a724137797d96a25 [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 5 got 0
I20240503 02:12:18.612460 5159 tablet_replica.cc:330] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:18.612893 5159 raft_consensus.cc:2229] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.613540 5159 raft_consensus.cc:2258] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.614598 5159 ts_tablet_manager.cc:1875] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.615379 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_96ec56aa54e84e89a80c091d6bcd5e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.615509 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 14fd4fb88436498999b5f339052ac0b6 (DEFAULT_TABLE table=default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.616045 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96ec56aa54e84e89a80c091d6bcd5e2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.617460 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 634beeb9d8394b1da8c4ad79ba7e26ac (table default.Table_28877a245811426292a39f20283988a2 [id=536a3d22f6e34248b7936ba2e7a05fd5]) successfully deleted
I20240503 02:12:18.618597 5103 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.618558 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 14fd4fb88436498999b5f339052ac0b6 (DEFAULT_TABLE table=default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.618559 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 (DEFAULT_TABLE table=default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.618786 5103 ts_tablet_manager.cc:1367] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240503 02:12:18.619067 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fd4fb88436498999b5f339052ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.619105 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ec8c0cdf74694faba53463bb1c5a4b0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.619326 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f72aa1636df46ce9eec9c69c2f70c49. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:18.619753 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec8c0cdf74694faba53463bb1c5a4b0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.619670 5103 raft_consensus.cc:348] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.619899 5103 raft_consensus.cc:374] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.619978 5103 raft_consensus.cc:729] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.620258 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 851756dc1a5942919c85967f83eca29a (DEFAULT_TABLE table=default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.621109 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 (DEFAULT_TABLE table=default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.622335 5103 consensus_queue.cc:260] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.626930 5103 ts_tablet_manager.cc:1398] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20240503 02:12:18.629599 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b487131ac3a842009b3fe54578ea091d consensus state has no leader
I20240503 02:12:18.630681 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_7941714137fb4383a724137797d96a25 [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 5 got 0
I20240503 02:12:18.630712 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 (DEFAULT_TABLE table=default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.631458 5103 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.631608 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f72aa1636df46ce9eec9c69c2f70c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.631621 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_eeccbe7767854540840b84af4d2820cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.632165 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeccbe7767854540840b84af4d2820cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.633561 5103 tablet_bootstrap.cc:654] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.636364 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 851756dc1a5942919c85967f83eca29a (DEFAULT_TABLE table=default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.637025 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851756dc1a5942919c85967f83eca29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.636943 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 851756dc1a5942919c85967f83eca29a (DEFAULT_TABLE table=default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.637274 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f (DEFAULT_TABLE table=default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920 [id=cad57d8578d2442984a701a12ebd16ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.637434 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851756dc1a5942919c85967f83eca29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.637538 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:18.638367 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d50112d374614b09ab8cdc93c129eac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.638957 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d50112d374614b09ab8cdc93c129eac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.640362 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 21e625e02b764ff2b7279b35dd10b08c
I20240503 02:12:18.641247 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_29408dd96a06477d99f95b091aa7f728"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.641817 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29408dd96a06477d99f95b091aa7f728 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.641803 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f (DEFAULT_TABLE table=default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920 [id=cad57d8578d2442984a701a12ebd16ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.642213 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a829c48a5c24d6d9e6329ac34151d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.643114 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 110c67212ec345028024da32968869ba
I20240503 02:12:18.643935 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_12bb031b9d404b97b3d144d9a596a2f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.644176 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 546afc74e6c54f8aa5f7c684484d610b (DEFAULT_TABLE table=default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.644488 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12bb031b9d404b97b3d144d9a596a2f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.645678 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f (DEFAULT_TABLE table=default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920 [id=cad57d8578d2442984a701a12ebd16ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.645836 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b7ddc725b90349e38301cfc29c21d848
I20240503 02:12:18.646142 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a829c48a5c24d6d9e6329ac34151d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.646087 4646 tablet_service.cc:1467] Processing CreateTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 (DEFAULT_TABLE table=default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.646303 4271 catalog_manager.cc:6554] ReplaceTablet: tablet b7ddc725b90349e38301cfc29c21d848 deleted and replaced by tablet 9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:18.646703 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d2c17788a5e148c4901c108226a3e05a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.647257 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2c17788a5e148c4901c108226a3e05a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.648074 4500 tablet_service.cc:1467] Processing CreateTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 (DEFAULT_TABLE table=default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.648545 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5117d4d59b4f0b87c32e8b27b085d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.649101 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.650451 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7b69392fa5f441049437639e143a7491
I20240503 02:12:18.651084 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet c126b8ef15c5432abf037e28a5eeba58 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.655589 4792 tablet_service.cc:1467] Processing CreateTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 (DEFAULT_TABLE table=default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.656244 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5117d4d59b4f0b87c32e8b27b085d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.658665 5106 tablet_bootstrap.cc:492] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:18.658850 5106 ts_tablet_manager.cc:1367] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.007s
I20240503 02:12:18.659780 5106 raft_consensus.cc:348] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.660079 5106 raft_consensus.cc:374] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.660187 5106 raft_consensus.cc:729] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.660504 5106 consensus_queue.cc:260] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.661124 5106 ts_tablet_manager.cc:1398] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:18.661422 5106 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:18.665300 5103 tablet_bootstrap.cc:492] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.665505 5103 ts_tablet_manager.cc:1367] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
I20240503 02:12:18.666357 5103 raft_consensus.cc:348] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.666589 5103 raft_consensus.cc:374] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.666700 5103 raft_consensus.cc:729] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.666954 5103 consensus_queue.cc:260] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.667263 4655 tablet_service.cc:1467] Processing CreateTablet for tablet a295609170a8482f8373dbf817b1eea1 (DEFAULT_TABLE table=default.Table_f396f63d98b5441d8f9ccc43949c520e [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.667596 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f57f3833a9240c88e0de1b486fae265 has no consensus state
I20240503 02:12:18.667719 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a295609170a8482f8373dbf817b1eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.667819 4508 tablet_service.cc:1467] Processing CreateTablet for tablet a295609170a8482f8373dbf817b1eea1 (DEFAULT_TABLE table=default.Table_f396f63d98b5441d8f9ccc43949c520e [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.668239 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a295609170a8482f8373dbf817b1eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.676601 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 (DEFAULT_TABLE table=default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.677120 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7daa5af6c8044178b9f84aaba4cca33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.678778 4508 tablet_service.cc:1467] Processing CreateTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 (DEFAULT_TABLE table=default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.679250 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7daa5af6c8044178b9f84aaba4cca33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.684417 5103 ts_tablet_manager.cc:1398] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.019s user 0.000s sys 0.003s
I20240503 02:12:18.684495 4791 tablet_service.cc:1467] Processing CreateTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 (DEFAULT_TABLE table=default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.684788 4793 tablet_service.cc:1467] Processing CreateTablet for tablet a295609170a8482f8373dbf817b1eea1 (DEFAULT_TABLE table=default.Table_f396f63d98b5441d8f9ccc43949c520e [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.685081 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7daa5af6c8044178b9f84aaba4cca33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.686358 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c126b8ef15c5432abf037e28a5eeba58
I20240503 02:12:18.686635 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a295609170a8482f8373dbf817b1eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.686924 4275 catalog_manager.cc:6554] ReplaceTablet: tablet c126b8ef15c5432abf037e28a5eeba58 deleted and replaced by tablet 7d5e627d818245e0bdeb16531499f7d5
I20240503 02:12:18.688475 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.689189 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.689826 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.697566 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.697579 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.698473 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.698732 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.698983 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ddc725b90349e38301cfc29c21d848 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f57f3833a9240c88e0de1b486fae265) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.699116 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.699837 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ddc725b90349e38301cfc29c21d848 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f57f3833a9240c88e0de1b486fae265) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.700498 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69392fa5f441049437639e143a7491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.700728 5103 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.701006 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ddc725b90349e38301cfc29c21d848 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f57f3833a9240c88e0de1b486fae265) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.701336 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69392fa5f441049437639e143a7491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.701614 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.701902 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69392fa5f441049437639e143a7491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.702911 5103 tablet_bootstrap.cc:654] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.703344 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fd4fb88436498999b5f339052ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.703977 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851756dc1a5942919c85967f83eca29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.704474 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5117d4d59b4f0b87c32e8b27b085d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.704977 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f72aa1636df46ce9eec9c69c2f70c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.705528 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a829c48a5c24d6d9e6329ac34151d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.706137 5106 tablet_bootstrap.cc:654] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.708720 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546afc74e6c54f8aa5f7c684484d610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.716006 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d3b1a905213545358364552f458c66cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.716621 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3b1a905213545358364552f458c66cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.716734 5114 ts_tablet_manager.cc:1888] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.716879 5114 log.cc:1196] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/a2dfd4c7a6f3483cace427cd87003535
I20240503 02:12:18.717284 5114 ts_tablet_manager.cc:1909] T a2dfd4c7a6f3483cace427cd87003535 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:18.719936 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet c126b8ef15c5432abf037e28a5eeba58 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d5e627d818245e0bdeb16531499f7d5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.720316 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ec8c0cdf74694faba53463bb1c5a4b0a" } modify_external_catalogs: true
I20240503 02:12:18.720518 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ec8c0cdf74694faba53463bb1c5a4b0a" } modify_external_catalogs: true
I20240503 02:12:18.720858 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_824bbb1347704b7db8daae6fdde91a26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.721359 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet c126b8ef15c5432abf037e28a5eeba58 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d5e627d818245e0bdeb16531499f7d5) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:18.721488 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_824bbb1347704b7db8daae6fdde91a26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.723546 4506 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 (DEFAULT_TABLE table=default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.723949 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c55f80a644cefa87b5769f15bb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.725311 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b69392fa5f441049437639e143a7491 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.726600 4813 raft_consensus.cc:1261] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:18.727213 5159 ts_tablet_manager.cc:1888] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:18.727356 5159 log.cc:1196] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/f22a4b8015354db493652d8f336a1265
I20240503 02:12:18.727470 5157 consensus_queue.cc:1035] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:18.730235 5141 raft_consensus.cc:482] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.730406 5141 raft_consensus.cc:504] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.731179 5141 leader_election.cc:288] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:18.731869 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:18.732097 4529 raft_consensus.cc:2454] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:18.732656 4746 leader_election.cc:302] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:18.733500 5141 raft_consensus.cc:2781] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:18.733625 5141 raft_consensus.cc:482] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:18.733747 5141 raft_consensus.cc:3037] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.734426 5159 ts_tablet_manager.cc:1909] T f22a4b8015354db493652d8f336a1265 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.735494 4506 tablet_service.cc:1467] Processing CreateTablet for tablet ced251fecca84866ab3ff15bd225e2b7 (DEFAULT_TABLE table=default.Table_28877a245811426292a39f20283988a2 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.735906 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced251fecca84866ab3ff15bd225e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.736514 5141 raft_consensus.cc:504] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.737854 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:18.738076 4529 raft_consensus.cc:3037] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.739976 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:18.741154 4441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:18.741547 4529 raft_consensus.cc:2454] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:18.741936 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 (DEFAULT_TABLE table=default.Table_e6d61608aa724f8daa53ca5585ab2947 [id=19642708612b48da91d318e497ba7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.742090 4746 leader_election.cc:302] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:18.742455 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e8c98d5ea84179a91fdbea8657d6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.742605 5159 tablet_replica.cc:330] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:18.743225 5159 raft_consensus.cc:2229] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:18.746579 5159 pending_rounds.cc:62] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:18.747426 5159 pending_rounds.cc:68] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420776364904448 op_type: NO_OP noop_request { }
I20240503 02:12:18.748421 5159 raft_consensus.cc:2866] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:18.748867 5159 raft_consensus.cc:2258] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.749763 5141 leader_election.cc:288] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:18.750039 5141 raft_consensus.cc:2781] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:18.750306 5141 raft_consensus.cc:686] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:18.750393 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1845b8fdfc7b488f9e5050933d9d4cdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.750695 5141 consensus_queue.cc:237] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:18.751041 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1845b8fdfc7b488f9e5050933d9d4cdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.751333 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 (DEFAULT_TABLE table=default.Table_e6d61608aa724f8daa53ca5585ab2947 [id=19642708612b48da91d318e497ba7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.751498 5157 raft_consensus.cc:482] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.751677 5157 raft_consensus.cc:504] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.752519 5157 leader_election.cc:288] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.752645 4783 tablet_service.cc:1467] Processing CreateTablet for tablet ced251fecca84866ab3ff15bd225e2b7 (DEFAULT_TABLE table=default.Table_28877a245811426292a39f20283988a2 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.753074 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced251fecca84866ab3ff15bd225e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.754086 5159 ts_tablet_manager.cc:1875] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.756160 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_00f06d9612844bcfba5ade0c41a86f77" } new_table_name: "default.Table_be08a6b601e14feb9ab4d23eb62484a6" modify_external_catalogs: true
I20240503 02:12:18.757931 4788 tablet_service.cc:1467] Processing CreateTablet for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (DEFAULT_TABLE table=default.Table_d9b602cd0e824ea185e468fd32acd15e [id=7048bc91b0b844f694937676dbc1026f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.758513 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe75d31e5da645be9ccdde4d0a4f9e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.763080 4792 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 (DEFAULT_TABLE table=default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.763629 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c55f80a644cefa87b5769f15bb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:18.764246 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc887bc4db1c4e268c4c9f23096d704c consensus state has no leader
I20240503 02:12:18.767113 5103 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:18.768190 5105 tablet_bootstrap.cc:492] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.768360 5105 ts_tablet_manager.cc:1367] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.182s user 0.001s sys 0.023s
I20240503 02:12:18.769150 5105 raft_consensus.cc:348] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.769398 5105 raft_consensus.cc:374] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.769479 5105 raft_consensus.cc:729] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.769827 5103 ts_tablet_manager.cc:1367] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.000s
I20240503 02:12:18.769739 5105 consensus_queue.cc:260] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.770699 5103 raft_consensus.cc:348] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.770731 4503 tablet_service.cc:1467] Processing CreateTablet for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (DEFAULT_TABLE table=default.Table_d9b602cd0e824ea185e468fd32acd15e [id=7048bc91b0b844f694937676dbc1026f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.770970 5103 raft_consensus.cc:374] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.771055 5103 raft_consensus.cc:729] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.771333 5103 consensus_queue.cc:260] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.775882 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet a2dfd4c7a6f3483cace427cd87003535 (table default.Table_06755c09ee504c3d8469e8b4d2acf8a3 [id=d6f6800254474aaeaf206d5d83bb78d6]) successfully deleted
I20240503 02:12:18.777858 5103 ts_tablet_manager.cc:1398] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:18.778013 5159 ts_tablet_manager.cc:1888] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:18.778136 5159 log.cc:1196] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b7ddc725b90349e38301cfc29c21d848
I20240503 02:12:18.778180 5114 tablet_replica.cc:330] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:18.778640 5114 raft_consensus.cc:2229] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.778947 5114 raft_consensus.cc:2258] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.780130 5159 ts_tablet_manager.cc:1909] T b7ddc725b90349e38301cfc29c21d848 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:18.783433 5105 ts_tablet_manager.cc:1398] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.015s user 0.000s sys 0.006s
I20240503 02:12:18.783859 5105 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:18.785635 5105 tablet_bootstrap.cc:654] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:18.788041 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 45939.2us
I20240503 02:12:18.793023 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_28877a245811426292a39f20283988a2" } new_table_name: "default.Table_4227e90304ee491ca298dabd17aa7b44" modify_external_catalogs: true
I20240503 02:12:18.796671 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:18.796994 4674 raft_consensus.cc:3037] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.797428 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.797628 4650 tablet_service.cc:1467] Processing CreateTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 (DEFAULT_TABLE table=default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.797794 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:18.799505 5114 ts_tablet_manager.cc:1875] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:18.799854 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6c3c676734a64db68d02daa66f4d8ee1" } new_table_name: "default.Table_5c5ea1ab22594d6b81fbd5b536231007" modify_external_catalogs: true
I20240503 02:12:18.802000 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (DEFAULT_TABLE table=default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.802517 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.807157 4648 tablet_service.cc:1467] Processing CreateTablet for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (DEFAULT_TABLE table=default.Table_d9b602cd0e824ea185e468fd32acd15e [id=7048bc91b0b844f694937676dbc1026f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.808423 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.808843 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3c55f80a644cefa87b5769f15bb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.810779 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (DEFAULT_TABLE table=default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.811235 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.811525 4640 tablet_service.cc:1467] Processing CreateTablet for tablet ced251fecca84866ab3ff15bd225e2b7 (DEFAULT_TABLE table=default.Table_28877a245811426292a39f20283988a2 [id=536a3d22f6e34248b7936ba2e7a05fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.812088 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (DEFAULT_TABLE table=default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.812647 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.813580 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:18.813835 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.814038 4974 raft_consensus.cc:482] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.814168 4974 raft_consensus.cc:504] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.814277 4783 tablet_service.cc:1467] Processing CreateTablet for tablet da12dd28e03f462b905d1844e2ad3340 (DEFAULT_TABLE table=default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.814637 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461a6b30ab59485ea1a2ef8ae9815e04" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.815358 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da12dd28e03f462b905d1844e2ad3340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.815443 4675 raft_consensus.cc:2454] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:18.816666 4649 tablet_service.cc:1467] Processing CreateTablet for tablet da12dd28e03f462b905d1844e2ad3340 (DEFAULT_TABLE table=default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.817433 4462 leader_election.cc:341] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.818957 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da12dd28e03f462b905d1844e2ad3340. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:18.819594 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc0e75639a4f4f168f9d833b090ae76a consensus state has no leader
I20240503 02:12:18.821774 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:18.821841 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48135062d92949dc8386c43cf103ced4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:18.822147 4529 raft_consensus.cc:2379] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
W20240503 02:12:18.822219 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ced251fecca84866ab3ff15bd225e2b7 consensus state has no leader
I20240503 02:12:18.823000 5157 raft_consensus.cc:482] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.822943 4508 tablet_service.cc:1467] Processing CreateTablet for tablet da12dd28e03f462b905d1844e2ad3340 (DEFAULT_TABLE table=default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.823143 5157 raft_consensus.cc:504] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.823828 5157 leader_election.cc:288] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.824524 5103 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:18.826910 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 (DEFAULT_TABLE table=default.Table_12bb031b9d404b97b3d144d9a596a2f5 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.827391 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc9fe2302024e5baa74d32941d70d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.827380 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 (DEFAULT_TABLE table=default.Table_e6d61608aa724f8daa53ca5585ab2947 [id=19642708612b48da91d318e497ba7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.824904 4974 leader_election.cc:288] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.828356 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effb2ce36a31477382865461d89917be" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:18.830032 4462 leader_election.cc:341] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.830350 5157 raft_consensus.cc:482] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.830531 5157 raft_consensus.cc:504] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.831234 5157 leader_election.cc:288] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.833802 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe75d31e5da645be9ccdde4d0a4f9e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.836056 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_824bbb1347704b7db8daae6fdde91a26" } new_table_name: "default.Table_5e9ba5798f4c4414826196e0761c2df9" modify_external_catalogs: true
I20240503 02:12:18.837144 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 (DEFAULT_TABLE table=default.Table_12bb031b9d404b97b3d144d9a596a2f5 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.843331 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa (DEFAULT_TABLE table=default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:18.843626 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a51563396614c54b980f0679e84a2b1 consensus state has no leader
I20240503 02:12:18.844226 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c9c3e0a1d400c94bfaefada2e9faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.845898 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced251fecca84866ab3ff15bd225e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.845103 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet c126b8ef15c5432abf037e28a5eeba58 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d5e627d818245e0bdeb16531499f7d5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:18.848335 4674 raft_consensus.cc:1261] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:18.850657 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa (DEFAULT_TABLE table=default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.850749 5157 consensus_queue.cc:1035] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:18.850867 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1fb4997ae544428a3067a3b9ae447dc" } modify_external_catalogs: true
I20240503 02:12:18.851137 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1fb4997ae544428a3067a3b9ae447dc" } modify_external_catalogs: true
I20240503 02:12:18.852373 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 (DEFAULT_TABLE table=default.Table_12bb031b9d404b97b3d144d9a596a2f5 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.853078 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8c833315113b4fafb24834feb6fc6649" } modify_external_catalogs: true
I20240503 02:12:18.853287 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8c833315113b4fafb24834feb6fc6649" } modify_external_catalogs: true
I20240503 02:12:18.855024 4608 leader_election.cc:302] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:18.855871 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25406a6f7c104671ac8b11de0c93b161" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.856858 5157 raft_consensus.cc:482] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:18.856904 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a38d73ea23ac44cab66f3cabba1f9fc2" } modify_external_catalogs: true
I20240503 02:12:18.857098 5157 raft_consensus.cc:504] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.857200 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a38d73ea23ac44cab66f3cabba1f9fc2" } modify_external_catalogs: true
I20240503 02:12:18.857237 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57f3833a9240c88e0de1b486fae265 (DEFAULT_TABLE table=default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.857832 5157 leader_election.cc:288] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:18.859525 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5e9ba5798f4c4414826196e0761c2df9" } modify_external_catalogs: true
I20240503 02:12:18.859759 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5e9ba5798f4c4414826196e0761c2df9" } modify_external_catalogs: true
W20240503 02:12:18.860055 4462 leader_election.cc:341] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:18.860302 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e8c98d5ea84179a91fdbea8657d6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.861461 4974 raft_consensus.cc:3037] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:18.866115 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b487131ac3a842009b3fe54578ea091d
I20240503 02:12:18.871769 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_21c620bc30de4de5b37ce3993ff24d1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.872494 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc9fe2302024e5baa74d32941d70d29. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:18.872642 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21c620bc30de4de5b37ce3993ff24d1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.876811 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_12bb031b9d404b97b3d144d9a596a2f5" } new_table_name: "default.Table_040a767183a44ec8a05884025834c555" modify_external_catalogs: true
I20240503 02:12:18.876753 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b803cbe542ac475bba57f562c9329076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.877394 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b803cbe542ac475bba57f562c9329076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.877620 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57f3833a9240c88e0de1b486fae265 (DEFAULT_TABLE table=default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.878093 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57f3833a9240c88e0de1b486fae265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.878705 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa (DEFAULT_TABLE table=default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.880228 4974 raft_consensus.cc:2726] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:18.880832 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f54d953fb24a3ea4bb21b6cf395edf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:18.881337 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57f3833a9240c88e0de1b486fae265 (DEFAULT_TABLE table=default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.883543 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ab3c55f80a644cefa87b5769f15bb7f0
W20240503 02:12:18.884150 4462 leader_election.cc:341] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:18.887109 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fe93cfdc45fd4972aa641dc189805745"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.887881 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe93cfdc45fd4972aa641dc189805745 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:18.892618 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bc9fe2302024e5baa74d32941d70d29 consensus state has no leader
I20240503 02:12:18.895897 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 851756dc1a5942919c85967f83eca29a
I20240503 02:12:18.897955 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 79aae500e9a842ed90357f4a65fd37ba (DEFAULT_TABLE table=default.Table_d50112d374614b09ab8cdc93c129eac2 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.897910 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ff1f8d575d8d42efb9bf4356accceefd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:18.898367 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet c126b8ef15c5432abf037e28a5eeba58 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:18.898555 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff1f8d575d8d42efb9bf4356accceefd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.899917 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_033290f38d39444aad4953c5f91ffaeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.900442 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_033290f38d39444aad4953c5f91ffaeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.900666 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a7065a46b193410db33240f20db15636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.901176 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7065a46b193410db33240f20db15636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.897994 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4fb41c047c2d4bf69f770a41417457a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.902451 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fb41c047c2d4bf69f770a41417457a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.907375 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c9c3e0a1d400c94bfaefada2e9faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.907797 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 79aae500e9a842ed90357f4a65fd37ba (DEFAULT_TABLE table=default.Table_d50112d374614b09ab8cdc93c129eac2 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.908672 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aae500e9a842ed90357f4a65fd37ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.908762 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aae500e9a842ed90357f4a65fd37ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.910012 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57f3833a9240c88e0de1b486fae265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.913389 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2a51563396614c54b980f0679e84a2b1
I20240503 02:12:18.924510 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet fc887bc4db1c4e268c4c9f23096d704c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:18.929536 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 79aae500e9a842ed90357f4a65fd37ba (DEFAULT_TABLE table=default.Table_d50112d374614b09ab8cdc93c129eac2 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.933305 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5456c87cd6d8459f9d35100bec3f8ffc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.933954 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5456c87cd6d8459f9d35100bec3f8ffc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.936326 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4eb187ba818c4f67a55aaeb3f44caae6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.936954 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eb187ba818c4f67a55aaeb3f44caae6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:18.942587 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4227e90304ee491ca298dabd17aa7b44" } modify_external_catalogs: true
I20240503 02:12:18.942807 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4227e90304ee491ca298dabd17aa7b44" } modify_external_catalogs: true
I20240503 02:12:18.952970 5105 tablet_bootstrap.cc:492] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.953153 5105 ts_tablet_manager.cc:1367] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.169s user 0.048s sys 0.038s
I20240503 02:12:18.954011 5105 raft_consensus.cc:348] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.954253 5105 raft_consensus.cc:374] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.954337 5105 raft_consensus.cc:729] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:18.954578 5105 consensus_queue.cc:260] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.955117 5105 ts_tablet_manager.cc:1398] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:18.955437 5105 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:18.961273 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fc887bc4db1c4e268c4c9f23096d704c
I20240503 02:12:18.961815 4270 catalog_manager.cc:6554] ReplaceTablet: tablet fc887bc4db1c4e268c4c9f23096d704c deleted and replaced by tablet 894b753b71a04193905a190da320a260
I20240503 02:12:18.962353 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:18.965982 5105 tablet_bootstrap.cc:654] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.966877 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6d61608aa724f8daa53ca5585ab2947" } new_table_name: "default.Table_a7538f13504946b6af3145d6c6282670" modify_external_catalogs: true
I20240503 02:12:18.970193 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fb193ca635ea47dc98041fb0c9671e82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:18.970903 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb193ca635ea47dc98041fb0c9671e82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:18.973784 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27e8c98d5ea84179a91fdbea8657d6e6 consensus state has no leader
I20240503 02:12:18.976747 4784 tablet_service.cc:1467] Processing CreateTablet for tablet d989197a5d6e42379df6d8add449a913 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.977339 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d989197a5d6e42379df6d8add449a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.977949 4643 tablet_service.cc:1467] Processing CreateTablet for tablet d989197a5d6e42379df6d8add449a913 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:18.986560 5105 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:18.751819 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e8c98d5ea84179a91fdbea8657d6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.988720 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe75d31e5da645be9ccdde4d0a4f9e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.989233 5105 ts_tablet_manager.cc:1367] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.001s
I20240503 02:12:18.990173 5105 raft_consensus.cc:348] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.990558 5105 raft_consensus.cc:374] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:18.990687 5105 raft_consensus.cc:729] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:18.990645 4462 leader_election.cc:341] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:18.990829 4462 leader_election.cc:302] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:18.991070 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aae500e9a842ed90357f4a65fd37ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.991020 5105 consensus_queue.cc:260] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:18.991389 5103 tablet_bootstrap.cc:654] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.991474 5157 raft_consensus.cc:2726] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:18.992066 5105 ts_tablet_manager.cc:1398] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:18.992476 5105 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:18.994153 5105 tablet_bootstrap.cc:654] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:18.996716 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57f3833a9240c88e0de1b486fae265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.997757 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d989197a5d6e42379df6d8add449a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.001072 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c9c3e0a1d400c94bfaefada2e9faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.004755 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.005103 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc9fe2302024e5baa74d32941d70d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.005319 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effb2ce36a31477382865461d89917be" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.006094 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25406a6f7c104671ac8b11de0c93b161" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.006724 4811 raft_consensus.cc:2454] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:19.007329 5103 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.007545 5103 ts_tablet_manager.cc:1367] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.183s user 0.010s sys 0.006s
I20240503 02:12:19.008519 5103 raft_consensus.cc:348] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.008751 5103 raft_consensus.cc:374] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.008857 5103 raft_consensus.cc:729] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.009102 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da12dd28e03f462b905d1844e2ad3340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.009197 5103 consensus_queue.cc:260] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.009851 5103 ts_tablet_manager.cc:1398] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:19.010334 5103 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.012061 5103 tablet_bootstrap.cc:654] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.017386 5103 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.017606 5103 ts_tablet_manager.cc:1367] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240503 02:12:19.018491 5103 raft_consensus.cc:348] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.018723 5103 raft_consensus.cc:374] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.018811 5103 raft_consensus.cc:729] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.019070 5103 consensus_queue.cc:260] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.019657 5103 ts_tablet_manager.cc:1398] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:19.020022 5103 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.021404 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f54d953fb24a3ea4bb21b6cf395edf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.021515 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4fb41c047c2d4bf69f770a41417457a1" } new_table_name: "default.Table_7f7a688514414e1cbb2d14173f3d1976" modify_external_catalogs: true
I20240503 02:12:19.021718 4810 raft_consensus.cc:2454] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:19.021925 5105 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:19.022114 5105 ts_tablet_manager.cc:1367] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
W20240503 02:12:19.022339 4462 leader_election.cc:341] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.022948 5105 raft_consensus.cc:348] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.023095 4462 leader_election.cc:302] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:19.023264 5105 raft_consensus.cc:374] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.023417 5105 raft_consensus.cc:729] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.023586 5157 raft_consensus.cc:2726] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.023746 5105 consensus_queue.cc:260] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.024356 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_85eb2a2a9c2b42d4bad6242f8e83006c" } new_table_name: "default.Table_2243856948a74aaab070be52ee0905a2" modify_external_catalogs: true
I20240503 02:12:19.024636 5105 ts_tablet_manager.cc:1398] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:19.025017 5105 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:19.026746 5105 tablet_bootstrap.cc:654] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.028821 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_5c37ffeee865425989dc0aa0cc41169a [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 2 got 0
I20240503 02:12:19.029058 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.032385 5103 tablet_bootstrap.cc:654] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:19.032771 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c823155b774e42a5a422994903eae167 consensus state has no leader
I20240503 02:12:19.034338 5157 raft_consensus.cc:482] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.034577 5157 raft_consensus.cc:504] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.035418 5157 leader_election.cc:288] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.037336 4462 leader_election.cc:302] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:19.037101 4272 catalog_manager.cc:5484] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:19.037712 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_7941714137fb4383a724137797d96a25 [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 5 got 0
I20240503 02:12:19.037940 5157 raft_consensus.cc:2781] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:19.038120 5157 raft_consensus.cc:482] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:19.038419 5157 raft_consensus.cc:3037] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.039088 4462 leader_election.cc:302] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:19.039495 4988 raft_consensus.cc:2781] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:19.039635 4988 raft_consensus.cc:482] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:19.039733 4988 raft_consensus.cc:3037] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.044706 4505 tablet_service.cc:1467] Processing CreateTablet for tablet d989197a5d6e42379df6d8add449a913 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.045217 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d989197a5d6e42379df6d8add449a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.045796 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7941714137fb4383a724137797d96a25" } new_table_name: "default.Table_7e4b66c1ce9a4b48b06d8c87e7fc4baa" modify_external_catalogs: true
I20240503 02:12:19.051149 4509 tablet_service.cc:1467] Processing CreateTablet for tablet c63499fb112c48238d6b84110e394779 (DEFAULT_TABLE table=default.Table_d2c17788a5e148c4901c108226a3e05a [id=01a00f4e2d344b4e817e920b106c6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.051607 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63499fb112c48238d6b84110e394779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.056178 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet f22a4b8015354db493652d8f336a1265 (table default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]) successfully deleted
I20240503 02:12:19.056187 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.056562 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0f72aa1636df46ce9eec9c69c2f70c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f72aa1636df46ce9eec9c69c2f70c49 already in progress: creating tablet
I20240503 02:12:19.056752 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5e627d818245e0bdeb16531499f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.056761 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 110c67212ec345028024da32968869ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110c67212ec345028024da32968869ba already in progress: creating tablet
I20240503 02:12:19.057153 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b7ddc725b90349e38301cfc29c21d848 (table default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]) successfully deleted
I20240503 02:12:19.058279 4974 raft_consensus.cc:482] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.066324 5157 raft_consensus.cc:504] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.067652 5157 leader_election.cc:288] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.068979 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eee1175b622840fe9e2ffb544bcfd426" } modify_external_catalogs: true
I20240503 02:12:19.069269 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eee1175b622840fe9e2ffb544bcfd426" } modify_external_catalogs: true
W20240503 02:12:19.072842 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7b69392fa5f441049437639e143a7491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b69392fa5f441049437639e143a7491 already in progress: creating tablet
W20240503 02:12:19.073340 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 21e625e02b764ff2b7279b35dd10b08c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e625e02b764ff2b7279b35dd10b08c already in progress: creating tablet
W20240503 02:12:19.073623 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c126b8ef15c5432abf037e28a5eeba58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c126b8ef15c5432abf037e28a5eeba58 already in progress: creating tablet
W20240503 02:12:19.073988 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because the tablet was not found. No further retry: Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa
W20240503 02:12:19.074679 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because the tablet was not found. No further retry: Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:19.077071 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_5c37ffeee865425989dc0aa0cc41169a [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 2 got 0
I20240503 02:12:19.079448 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0e75639a4f4f168f9d833b090ae76a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:19.084514 5115 ts_tablet_manager.cc:1888] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.086458 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:19.087253 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.087896 5106 tablet_bootstrap.cc:492] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:19.088168 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b741361e02de4b5aaed514fe7cd99f58" } modify_external_catalogs: true
I20240503 02:12:19.088445 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b741361e02de4b5aaed514fe7cd99f58" } modify_external_catalogs: true
I20240503 02:12:19.093295 4974 raft_consensus.cc:504] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.094064 4974 leader_election.cc:288] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.094843 4791 tablet_service.cc:1467] Processing CreateTablet for tablet c88eaba979394b449d96ee956c3f28a8 (DEFAULT_TABLE table=default.Table_d3b1a905213545358364552f458c66cd [id=62c39d7fbcb3410994f6599d86741094]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.095305 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 851756dc1a5942919c85967f83eca29a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.094909 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25406a6f7c104671ac8b11de0c93b161" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:19.095988 4810 raft_consensus.cc:3037] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.096412 5115 log.cc:1196] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/88f54d953fb24a3ea4bb21b6cf395edf
I20240503 02:12:19.097584 4648 tablet_service.cc:1467] Processing CreateTablet for tablet c63499fb112c48238d6b84110e394779 (DEFAULT_TABLE table=default.Table_d2c17788a5e148c4901c108226a3e05a [id=01a00f4e2d344b4e817e920b106c6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.099346 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.091444 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.100890 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f556497cf91b4b6cb71095aceb0c3a58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.101382 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7f7a688514414e1cbb2d14173f3d1976" } modify_external_catalogs: true
W20240503 02:12:19.101600 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f556497cf91b4b6cb71095aceb0c3a58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.101872 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 2a51563396614c54b980f0679e84a2b1 (DEFAULT_TABLE table=default.Table_824bbb1347704b7db8daae6fdde91a26 [id=0a6dd433bee8497395826808bba7774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.099248 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.092892 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0e75639a4f4f168f9d833b090ae76a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.101048 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b2b7945a22d649119846e9c546ddc27b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.102900 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c63499fb112c48238d6b84110e394779 (DEFAULT_TABLE table=default.Table_d2c17788a5e148c4901c108226a3e05a [id=01a00f4e2d344b4e817e920b106c6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.103083 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7e4b66c1ce9a4b48b06d8c87e7fc4baa" } new_table_name: "default.Table_8278654b5129427a9721eed448fc1758" modify_external_catalogs: true
I20240503 02:12:19.101598 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7f7a688514414e1cbb2d14173f3d1976" } modify_external_catalogs: true
I20240503 02:12:19.103322 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:19.103107 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2b7945a22d649119846e9c546ddc27b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.105502 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fe5117d4d59b4f0b87c32e8b27b085d9
I20240503 02:12:19.108069 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5e627d818245e0bdeb16531499f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.108201 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet b487131ac3a842009b3fe54578ea091d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.108228 4782 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 (DEFAULT_TABLE table=default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.109167 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c07c7cbb743d4a8e9610281a00f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.109421 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88eaba979394b449d96ee956c3f28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.119251 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_21c620bc30de4de5b37ce3993ff24d1a" } modify_external_catalogs: true
I20240503 02:12:19.119550 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_21c620bc30de4de5b37ce3993ff24d1a" } modify_external_catalogs: true
I20240503 02:12:19.123168 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet fc887bc4db1c4e268c4c9f23096d704c with delete_type TABLET_DATA_DELETED (replaced by tablet 894b753b71a04193905a190da320a260) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.123467 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (DEFAULT_TABLE table=default.Table_b803cbe542ac475bba57f562c9329076 [id=b4c01b578b424aa2871f82e607dba76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.123627 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c823155b774e42a5a422994903eae167
I20240503 02:12:19.124128 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46560ac6d94943d2a8fb3b3904f2b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.126704 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4967
I20240503 02:12:19.132037 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet bf9c07c7cbb743d4a8e9610281a00f94
I20240503 02:12:19.135140 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63499fb112c48238d6b84110e394779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.137288 4810 raft_consensus.cc:2454] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:19.137987 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.138062 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cced2ab4f3d94350970ba6daf24e51b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.137883 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1210a1217247b9801226055e2f2195 (DEFAULT_TABLE table=default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.138891 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cced2ab4f3d94350970ba6daf24e51b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.139672 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1210a1217247b9801226055e2f2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.140641 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_621dd8b829b34a0eafce9326b41c3a25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.141209 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_621dd8b829b34a0eafce9326b41c3a25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.142410 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4489651bcc554f9ca32e9f7a9629b9ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.142887 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4489651bcc554f9ca32e9f7a9629b9ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.144996 4785 tablet_service.cc:1467] Processing CreateTablet for tablet c823155b774e42a5a422994903eae167 (DEFAULT_TABLE table=default.Table_4fb41c047c2d4bf69f770a41417457a1 [id=7abc43f4f4eb42828defca24da288002]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.145449 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c823155b774e42a5a422994903eae167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.146857 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_535d1456ce3640c6a7e33dbc4b2c305c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.147558 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_535d1456ce3640c6a7e33dbc4b2c305c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.149979 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f54d953fb24a3ea4bb21b6cf395edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.150364 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 25406a6f7c104671ac8b11de0c93b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.150509 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.150655 5159 tablet_replica.cc:330] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:19.150876 5159 raft_consensus.cc:2229] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.151016 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53dad4e45a184dc1ae4a82021a48e9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.151067 5159 raft_consensus.cc:2258] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.153084 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet fed654900a11471eaf6f7a198f8a985b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.153398 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.154736 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 387a596cfe654aa99e83d070b7884e6b (DEFAULT_TABLE table=default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.155351 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a596cfe654aa99e83d070b7884e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.156373 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c2a0d32aa38141e8bd72a2e2293843c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.157146 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2a0d32aa38141e8bd72a2e2293843c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.158380 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 64cdb861bc9f448698e243718136c303 (DEFAULT_TABLE table=default.Table_5456c87cd6d8459f9d35100bec3f8ffc [id=6fb8f01b6f4d46f883d05d680f5c0ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.159015 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cdb861bc9f448698e243718136c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.161816 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 (DEFAULT_TABLE table=default.Table_fe93cfdc45fd4972aa641dc189805745 [id=5108d90cc915403ba879db13f6a9f1be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.162250 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4064ff59de474e0db3a578eae10c3fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.165432 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (DEFAULT_TABLE table=default.Table_fb193ca635ea47dc98041fb0c9671e82 [id=5765715dd8e74bd5b16aa04cbc596b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.165956 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd402c0eb9e4cf49dc722f2dd214e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.167099 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8fb97a4f46484d98a7e0d2d19788c5fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.167943 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fb97a4f46484d98a7e0d2d19788c5fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.169313 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b8cf4144fb1d4392856c92576ed4dcb1" } modify_external_catalogs: true
I20240503 02:12:19.169551 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b8cf4144fb1d4392856c92576ed4dcb1" } modify_external_catalogs: true
I20240503 02:12:19.170390 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 894b753b71a04193905a190da320a260 (DEFAULT_TABLE table=default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.170928 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894b753b71a04193905a190da320a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.173310 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6ee3f7d7e571413dba0fe216678bf70d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.173655 5159 ts_tablet_manager.cc:1875] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.179916 5159 ts_tablet_manager.cc:1888] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.180090 5159 log.cc:1196] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/88f54d953fb24a3ea4bb21b6cf395edf
I20240503 02:12:19.180543 5159 ts_tablet_manager.cc:1909] T 88f54d953fb24a3ea4bb21b6cf395edf P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:19.181821 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ee3f7d7e571413dba0fe216678bf70d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.182766 5159 tablet_replica.cc:330] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:19.183037 5159 raft_consensus.cc:2229] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.183293 5159 raft_consensus.cc:2258] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.178135 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d9b602cd0e824ea185e468fd32acd15e" } new_table_name: "default.Table_b43cd816d9de4772af938d5184d05ce4" modify_external_catalogs: true
I20240503 02:12:19.184309 5159 ts_tablet_manager.cc:1875] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.186946 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c0626018603746ea91e3a2d6cfc37ef8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.194506 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 546afc74e6c54f8aa5f7c684484d610b
I20240503 02:12:19.197885 4791 tablet_service.cc:1467] Processing CreateTablet for tablet c0b9003f248b47a9952e6f995887ae0c (DEFAULT_TABLE table=default.Table_ff1f8d575d8d42efb9bf4356accceefd [id=4dbc0d237a3d42aa8fe439bfeec4d668]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.198984 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b9003f248b47a9952e6f995887ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.202821 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe75d31e5da645be9ccdde4d0a4f9e05 consensus state has no leader
W20240503 02:12:19.204660 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0626018603746ea91e3a2d6cfc37ef8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.209525 5105 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:19.209735 5105 ts_tablet_manager.cc:1367] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.185s user 0.012s sys 0.020s
I20240503 02:12:19.210629 5105 raft_consensus.cc:348] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.210853 5105 raft_consensus.cc:374] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.210938 5105 raft_consensus.cc:729] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.211215 5105 consensus_queue.cc:260] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.211820 5105 ts_tablet_manager.cc:1398] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:19.214560 5159 ts_tablet_manager.cc:1888] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.215368 5159 log.cc:1196] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:19.216115 5159 ts_tablet_manager.cc:1909] T 25406a6f7c104671ac8b11de0c93b161 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:19.219705 4462 leader_election.cc:341] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.220238 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.220337 4988 raft_consensus.cc:504] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.221614 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 461a6b30ab59485ea1a2ef8ae9815e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.223376 4504 tablet_service.cc:1467] Processing CreateTablet for tablet c88eaba979394b449d96ee956c3f28a8 (DEFAULT_TABLE table=default.Table_d3b1a905213545358364552f458c66cd [id=62c39d7fbcb3410994f6599d86741094]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.228108 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1210a1217247b9801226055e2f2195 (DEFAULT_TABLE table=default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.232980 5177 raft_consensus.cc:482] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.233211 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet b487131ac3a842009b3fe54578ea091d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.233165 5177 raft_consensus.cc:504] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.234215 5177 leader_election.cc:288] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:19.236668 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ad0ceaed9ff641a8815b73da0ae0e02c" } new_table_name: "default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae" modify_external_catalogs: true
I20240503 02:12:19.238448 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b803cbe542ac475bba57f562c9329076" } new_table_name: "default.Table_b8a86d9e96be4890bad78b29e651918a" modify_external_catalogs: true
I20240503 02:12:19.239933 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (DEFAULT_TABLE table=default.Table_b803cbe542ac475bba57f562c9329076 [id=b4c01b578b424aa2871f82e607dba76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.240551 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:19.240700 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 136640us
I20240503 02:12:19.233793 4988 leader_election.cc:288] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.247427 5043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:19.253319 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ea1084304449e895be230d86471a9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:19.252646 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 2a51563396614c54b980f0679e84a2b1 (DEFAULT_TABLE table=default.Table_5e9ba5798f4c4414826196e0761c2df9 [id=0a6dd433bee8497395826808bba7774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.254201 4529 raft_consensus.cc:1261] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:19.255121 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a51563396614c54b980f0679e84a2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.257318 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 64cdb861bc9f448698e243718136c303 (DEFAULT_TABLE table=default.Table_5456c87cd6d8459f9d35100bec3f8ffc [id=6fb8f01b6f4d46f883d05d680f5c0ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.257413 5177 consensus_queue.cc:1035] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:19.256613 4462 leader_election.cc:341] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.258255 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1210a1217247b9801226055e2f2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.258457 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 894b753b71a04193905a190da320a260 (DEFAULT_TABLE table=default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.102203 4653 tablet_service.cc:1467] Processing CreateTablet for tablet c88eaba979394b449d96ee956c3f28a8 (DEFAULT_TABLE table=default.Table_d3b1a905213545358364552f458c66cd [id=62c39d7fbcb3410994f6599d86741094]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.260810 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88eaba979394b449d96ee956c3f28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.260903 5103 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.260090 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 2a51563396614c54b980f0679e84a2b1 (DEFAULT_TABLE table=default.Table_5e9ba5798f4c4414826196e0761c2df9 [id=0a6dd433bee8497395826808bba7774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.261101 5103 ts_tablet_manager.cc:1367] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.241s user 0.045s sys 0.048s
W20240503 02:12:19.261344 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46560ac6d94943d2a8fb3b3904f2b5b6 consensus state has no leader
I20240503 02:12:19.261988 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.262671 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63499fb112c48238d6b84110e394779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.263942 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 (DEFAULT_TABLE table=default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.264039 4649 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 (DEFAULT_TABLE table=default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.264757 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_040a767183a44ec8a05884025834c555" } new_table_name: "default.Table_422f8213fa2e4d269fc4499098056f64" modify_external_catalogs: true
I20240503 02:12:19.265017 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46560ac6d94943d2a8fb3b3904f2b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.271087 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 851756dc1a5942919c85967f83eca29a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.271175 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet b487131ac3a842009b3fe54578ea091d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.271116 5106 ts_tablet_manager.cc:1367] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.427s user 0.007s sys 0.012s
I20240503 02:12:19.271471 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (DEFAULT_TABLE table=default.Table_b803cbe542ac475bba57f562c9329076 [id=b4c01b578b424aa2871f82e607dba76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.273878 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1210a1217247b9801226055e2f2195 (DEFAULT_TABLE table=default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.275043 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bc9fe2302024e5baa74d32941d70d29 consensus state has no leader
I20240503 02:12:19.275357 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet fc887bc4db1c4e268c4c9f23096d704c with delete_type TABLET_DATA_DELETED (replaced by tablet 894b753b71a04193905a190da320a260) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.275627 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.276059 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25406a6f7c104671ac8b11de0c93b161" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:19.277510 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 (DEFAULT_TABLE table=default.Table_fe93cfdc45fd4972aa641dc189805745 [id=5108d90cc915403ba879db13f6a9f1be]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.278931 4647 consensus_meta.cc:287] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.567s user 0.005s sys 0.000s
I20240503 02:12:19.279561 4490 tablet_service.cc:1467] Processing CreateTablet for tablet c0b9003f248b47a9952e6f995887ae0c (DEFAULT_TABLE table=default.Table_ff1f8d575d8d42efb9bf4356accceefd [id=4dbc0d237a3d42aa8fe439bfeec4d668]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.279877 4655 tablet_service.cc:1467] Processing CreateTablet for tablet c0b9003f248b47a9952e6f995887ae0c (DEFAULT_TABLE table=default.Table_ff1f8d575d8d42efb9bf4356accceefd [id=4dbc0d237a3d42aa8fe439bfeec4d668]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.280099 4528 raft_consensus.cc:2454] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:19.280759 4746 leader_election.cc:302] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:19.281148 5177 raft_consensus.cc:2781] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:19.281247 5177 raft_consensus.cc:482] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:19.281340 5177 raft_consensus.cc:3037] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.282929 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f54d953fb24a3ea4bb21b6cf395edf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:19.283792 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 (DEFAULT_TABLE table=default.Table_fe93cfdc45fd4972aa641dc189805745 [id=5108d90cc915403ba879db13f6a9f1be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.284134 5177 raft_consensus.cc:504] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.285615 5106 raft_consensus.cc:348] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.287976 5177 leader_election.cc:288] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:19.285317 4508 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 (DEFAULT_TABLE table=default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.288763 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3ae28439fd6b46438f67fd812635c2f4" } modify_external_catalogs: true
I20240503 02:12:19.289022 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3ae28439fd6b46438f67fd812635c2f4" } modify_external_catalogs: true
I20240503 02:12:19.289894 4674 raft_consensus.cc:1261] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:19.292068 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (DEFAULT_TABLE table=default.Table_fb193ca635ea47dc98041fb0c9671e82 [id=5765715dd8e74bd5b16aa04cbc596b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.293295 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed654900a11471eaf6f7a198f8a985b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.293579 4810 raft_consensus.cc:2454] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:19.293776 4988 raft_consensus.cc:482] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.293965 4988 raft_consensus.cc:504] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.294888 4988 leader_election.cc:288] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.295153 4462 leader_election.cc:302] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:19.295570 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 52491us
I20240503 02:12:19.295750 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 894b753b71a04193905a190da320a260 (DEFAULT_TABLE table=default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.295878 4462 leader_election.cc:302] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:19.296108 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 387a596cfe654aa99e83d070b7884e6b (DEFAULT_TABLE table=default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.296219 4462 leader_election.cc:341] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 88f54d953fb24a3ea4bb21b6cf395edf
I20240503 02:12:19.296406 4608 leader_election.cc:302] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:19.297035 4462 leader_election.cc:341] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.297323 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110c67212ec345028024da32968869ba" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:19.297444 4462 leader_election.cc:341] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:19.297588 4462 leader_election.cc:302] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:19.297957 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet fc887bc4db1c4e268c4c9f23096d704c with delete_type TABLET_DATA_DELETED (replaced by tablet 894b753b71a04193905a190da320a260) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.298475 5191 raft_consensus.cc:2726] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.298655 5192 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.299147 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.299732 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.302129 5191 raft_consensus.cc:2726] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.302453 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 387a596cfe654aa99e83d070b7884e6b (DEFAULT_TABLE table=default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.300662 5177 consensus_queue.cc:1035] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:12:19.303664 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ea1084304449e895be230d86471a9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:19.303952 4491 tablet_service.cc:1467] Processing CreateTablet for tablet c823155b774e42a5a422994903eae167 (DEFAULT_TABLE table=default.Table_4fb41c047c2d4bf69f770a41417457a1 [id=7abc43f4f4eb42828defca24da288002]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.304968 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 851756dc1a5942919c85967f83eca29a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.307659 4988 raft_consensus.cc:2781] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:19.315985 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cdb861bc9f448698e243718136c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.317770 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7d5e627d818245e0bdeb16531499f7d5
I20240503 02:12:19.318933 4528 raft_consensus.cc:3037] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.322878 4528 raft_consensus.cc:2454] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:19.323906 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24737f8a8ca74996b9f54d49d12b9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.328737 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a596cfe654aa99e83d070b7884e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.330099 4746 leader_election.cc:302] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:19.330485 5141 raft_consensus.cc:2781] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:19.331912 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a15f08dd5c104b5295a3ebf6cd420b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.332618 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a15f08dd5c104b5295a3ebf6cd420b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.336155 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a7538f13504946b6af3145d6c6282670" } modify_external_catalogs: true
I20240503 02:12:19.336447 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a7538f13504946b6af3145d6c6282670" } modify_external_catalogs: true
I20240503 02:12:19.338238 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88eaba979394b449d96ee956c3f28a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.349311 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:19.355089 5141 raft_consensus.cc:686] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:19.355499 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c823155b774e42a5a422994903eae167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.356069 5141 consensus_queue.cc:237] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.358814 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b2b7945a22d649119846e9c546ddc27b" } modify_external_catalogs: true
I20240503 02:12:19.359481 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b2b7945a22d649119846e9c546ddc27b" } modify_external_catalogs: true
I20240503 02:12:19.359987 4988 raft_consensus.cc:686] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:19.363708 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3d4325dad27b49bfa8a8e38c55372f6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.364390 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d4325dad27b49bfa8a8e38c55372f6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.365809 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b9003f248b47a9952e6f995887ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.366593 4988 consensus_queue.cc:237] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.370098 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd402c0eb9e4cf49dc722f2dd214e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.372757 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0414bb3004e94f78a6ff888c2ef0cb8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.373418 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0414bb3004e94f78a6ff888c2ef0cb8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.373450 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1210a1217247b9801226055e2f2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.376252 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b6ddecc9e022479595b1ce5c50323184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.376451 5106 raft_consensus.cc:374] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:19.377030 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6ddecc9e022479595b1ce5c50323184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.378043 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ea1084304449e895be230d86471a9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:19.378525 4974 raft_consensus.cc:482] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.378726 4974 raft_consensus.cc:504] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.379532 4974 leader_election.cc:288] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.379734 4974 raft_consensus.cc:2781] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:19.379688 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9166690fbabf48e0b6a2cafbbb312ccd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.379848 4974 raft_consensus.cc:482] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:19.379935 4974 raft_consensus.cc:3037] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:19.380208 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9166690fbabf48e0b6a2cafbbb312ccd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.380713 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a51563396614c54b980f0679e84a2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.381232 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 64cdb861bc9f448698e243718136c303 (DEFAULT_TABLE table=default.Table_5456c87cd6d8459f9d35100bec3f8ffc [id=6fb8f01b6f4d46f883d05d680f5c0ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.381351 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d3b1a905213545358364552f458c66cd" } new_table_name: "default.Table_e66e7c44c2784f97bc80c180112e3e11" modify_external_catalogs: true
I20240503 02:12:19.381682 4636 tablet_service.cc:1467] Processing CreateTablet for tablet c823155b774e42a5a422994903eae167 (DEFAULT_TABLE table=default.Table_4fb41c047c2d4bf69f770a41417457a1 [id=7abc43f4f4eb42828defca24da288002]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.381826 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4064ff59de474e0db3a578eae10c3fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.382263 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (DEFAULT_TABLE table=default.Table_fb193ca635ea47dc98041fb0c9671e82 [id=5765715dd8e74bd5b16aa04cbc596b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.382684 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110c67212ec345028024da32968869ba" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:19.383791 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bba46dea649d4895bacf0532ffeb8a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.386927 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bba46dea649d4895bacf0532ffeb8a7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.387878 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.389544 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894b753b71a04193905a190da320a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.391142 5106 raft_consensus.cc:729] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.392084 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.394443 5106 consensus_queue.cc:260] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.394971 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a596cfe654aa99e83d070b7884e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.395264 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894b753b71a04193905a190da320a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.401883 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5e627d818245e0bdeb16531499f7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.401937 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1845b8fdfc7b488f9e5050933d9d4cdf" } modify_external_catalogs: true
I20240503 02:12:19.402148 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1845b8fdfc7b488f9e5050933d9d4cdf" } modify_external_catalogs: true
I20240503 02:12:19.403033 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46560ac6d94943d2a8fb3b3904f2b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.403231 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c823155b774e42a5a422994903eae167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.404181 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a51563396614c54b980f0679e84a2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.404386 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd402c0eb9e4cf49dc722f2dd214e09. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.405372 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c88eaba979394b449d96ee956c3f28a8 consensus state has no leader
I20240503 02:12:19.405851 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fb193ca635ea47dc98041fb0c9671e82" } new_table_name: "default.Table_5db43ecab50c42d6bd7cea5b311d5f95" modify_external_catalogs: true
I20240503 02:12:19.406049 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b9003f248b47a9952e6f995887ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.406574 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cdb861bc9f448698e243718136c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.406878 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c07c7cbb743d4a8e9610281a00f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.375231 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ea1084304449e895be230d86471a9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:19.410287 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c00f7cc799d34540b77a581d4a9a4561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.410717 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet acdefb33fe6e4a9893dd66f6252395d7
W20240503 02:12:19.410907 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c00f7cc799d34540b77a581d4a9a4561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.407918 4974 raft_consensus.cc:504] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.394104 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.411518 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4064ff59de474e0db3a578eae10c3fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.415709 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c07c7cbb743d4a8e9610281a00f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.420909 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8925d918e3de4d60b2f13a0c0e06f5be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.421880 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8925d918e3de4d60b2f13a0c0e06f5be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.424428 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 461a6b30ab59485ea1a2ef8ae9815e04 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:19.424743 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c88eaba979394b449d96ee956c3f28a8 consensus state has no leader
I20240503 02:12:19.329012 5103 raft_consensus.cc:348] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.429162 5103 raft_consensus.cc:374] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.432231 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53dad4e45a184dc1ae4a82021a48e9be. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.434530 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6cd402c0eb9e4cf49dc722f2dd214e09 consensus state has no leader
I20240503 02:12:19.438187 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24737f8a8ca74996b9f54d49d12b9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.432461 4974 leader_election.cc:288] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.438737 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet fed654900a11471eaf6f7a198f8a985b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.440026 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 110c67212ec345028024da32968869ba from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.441577 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ffd19b3ced6f4f58b7bc18123f9246c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.442209 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffd19b3ced6f4f58b7bc18123f9246c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.442610 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.442950 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53dad4e45a184dc1ae4a82021a48e9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.443068 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24737f8a8ca74996b9f54d49d12b9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.443862 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0ccc60c229e74e9197d025a8c631a76c" } modify_external_catalogs: true
I20240503 02:12:19.444140 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0ccc60c229e74e9197d025a8c631a76c" } modify_external_catalogs: true
I20240503 02:12:19.444944 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_06ee7797a50e4886b1d001c0538b9d50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.445614 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ee7797a50e4886b1d001c0538b9d50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.448083 5103 raft_consensus.cc:729] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.457058 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 003c9c3e0a1d400c94bfaefada2e9faa from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.460855 5105 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:19.462899 5105 tablet_bootstrap.cc:654] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.469293 5106 ts_tablet_manager.cc:1398] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.195s user 0.002s sys 0.006s
I20240503 02:12:19.474601 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ff1f8d575d8d42efb9bf4356accceefd" } new_table_name: "default.Table_3b9cdd4b8f9e4449aed0a52093a299b8" modify_external_catalogs: true
I20240503 02:12:19.475625 5103 consensus_queue.cc:260] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.475823 5159 tablet_replica.cc:330] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:19.476035 5159 raft_consensus.cc:2229] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.476188 5159 raft_consensus.cc:2258] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.476794 5159 ts_tablet_manager.cc:1875] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.482776 5103 ts_tablet_manager.cc:1398] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.215s user 0.007s sys 0.000s
I20240503 02:12:19.483764 5114 ts_tablet_manager.cc:1888] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.484421 5103 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.485955 5114 log.cc:1196] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/634beeb9d8394b1da8c4ad79ba7e26ac
I20240503 02:12:19.486340 5159 ts_tablet_manager.cc:1888] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.486495 5159 log.cc:1196] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:19.486815 5114 ts_tablet_manager.cc:1909] T 634beeb9d8394b1da8c4ad79ba7e26ac P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:19.486840 5159 ts_tablet_manager.cc:1909] T fed654900a11471eaf6f7a198f8a985b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:19.488250 5114 tablet_replica.cc:330] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:19.488327 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cced2ab4f3d94350970ba6daf24e51b4" } new_table_name: "default.Table_070e3cfe611e43baaec51dd9b0c1d7e9" modify_external_catalogs: true
I20240503 02:12:19.490094 5114 raft_consensus.cc:2229] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:19.490689 5114 pending_rounds.cc:62] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:19.490868 5114 pending_rounds.cc:68] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420782063316992 op_type: NO_OP noop_request { }
I20240503 02:12:19.491145 5114 raft_consensus.cc:2866] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:19.491353 5114 raft_consensus.cc:2258] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:19.491688 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 194228us
I20240503 02:12:19.492331 5115 ts_tablet_manager.cc:1909] T 88f54d953fb24a3ea4bb21b6cf395edf P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:19.495379 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet ced251fecca84866ab3ff15bd225e2b7 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.495589 5190 raft_consensus.cc:482] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.496110 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5c37ffeee865425989dc0aa0cc41169a" } new_table_name: "default.Table_8a018bc2b9e84350b29ef6447a8bb965" modify_external_catalogs: true
I20240503 02:12:19.501411 5190 raft_consensus.cc:504] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.501662 5106 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:19.503861 5159 tablet_replica.cc:330] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:19.504382 5159 raft_consensus.cc:2229] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:19.505195 4528 raft_consensus.cc:3037] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.506157 5190 leader_election.cc:288] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:19.508019 5206 consensus_queue.cc:1159] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f4f9f079230a468f90e5e3a58f254b91" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240503 02:12:19.508899 5114 ts_tablet_manager.cc:1875] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.514547 5159 pending_rounds.cc:62] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:19.514647 5159 pending_rounds.cc:68] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420782031519744 op_type: NO_OP noop_request { }
I20240503 02:12:19.515950 5159 raft_consensus.cc:2866] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:19.516161 5159 raft_consensus.cc:2258] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.516585 5105 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:19.517236 5105 ts_tablet_manager.cc:1367] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.001s
I20240503 02:12:19.520560 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4eb187ba818c4f67a55aaeb3f44caae6" } modify_external_catalogs: true
I20240503 02:12:19.521070 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4eb187ba818c4f67a55aaeb3f44caae6" } modify_external_catalogs: true
I20240503 02:12:19.522552 5159 ts_tablet_manager.cc:1875] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.521761 5105 raft_consensus.cc:348] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.523861 5105 raft_consensus.cc:374] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.525509 5103 tablet_bootstrap.cc:654] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.524453 5105 raft_consensus.cc:729] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:19.535055 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 440184us
I20240503 02:12:19.539445 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 30) took 1793 ms. Trace:
I20240503 02:12:19.540308 5115 rpcz_store.cc:276] 0503 02:12:17.743530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.743852 (+ 322us) service_pool.cc:225] Handling call
0503 02:12:18.502231 (+758379us) ts_tablet_manager.cc:1120] Deleting tablet 88f54d953fb24a3ea4bb21b6cf395edf
0503 02:12:18.521454 (+ 19223us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:18.708290 (+186836us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.000871 (+292581us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.537204 (+536333us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":262,"tablet-delete.queue_time_us":737343}
I20240503 02:12:19.544121 5115 tablet_replica.cc:330] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:19.544967 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.545063 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.545606 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 546afc74e6c54f8aa5f7c684484d610b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.545650 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.545997 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.546641 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fd4fb88436498999b5f339052ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.546850 5115 raft_consensus.cc:2229] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.548179 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:19.538208 5105 consensus_queue.cc:260] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.549027 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a210808781ce413f96c49311edbe1415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.549787 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a210808781ce413f96c49311edbe1415 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:19.554256 4746 leader_election.cc:341] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:19.555091 4746 leader_election.cc:341] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.557947 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b" } modify_external_catalogs: true
I20240503 02:12:19.558223 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b" } modify_external_catalogs: true
I20240503 02:12:19.560787 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.561908 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fd4fb88436498999b5f339052ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.562177 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 546afc74e6c54f8aa5f7c684484d610b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:19.562891 5202 consensus_queue.cc:1035] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:12:19.566617 5115 raft_consensus.cc:2258] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.568336 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:19.568620 4462 leader_election.cc:341] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.562920 5105 ts_tablet_manager.cc:1398] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.044s user 0.002s sys 0.001s
W20240503 02:12:19.569257 4462 leader_election.cc:341] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.569720 5105 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:19.569438 4462 leader_election.cc:302] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:19.570413 5190 raft_consensus.cc:2726] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.574182 5105 tablet_bootstrap.cc:654] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:19.575716 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f40706400f114def9e14e6d1d7114962 consensus state has no leader
W20240503 02:12:19.576557 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0b9003f248b47a9952e6f995887ae0c consensus state has no leader
I20240503 02:12:19.578615 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.583684 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_5c37ffeee865425989dc0aa0cc41169a [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 2 got 0
I20240503 02:12:19.586792 5192 tablet.cc:1717] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:19.590948 5114 ts_tablet_manager.cc:1888] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:19.591428 5114 log.cc:1196] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:19.591922 5114 ts_tablet_manager.cc:1909] T 25406a6f7c104671ac8b11de0c93b161 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:19.593043 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 51) took 1074 ms. Trace:
I20240503 02:12:19.598485 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 3 got 0
I20240503 02:12:19.599416 5114 rpcz_store.cc:276] 0503 02:12:18.518887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.535114 (+ 16227us) service_pool.cc:225] Handling call
0503 02:12:19.488158 (+953044us) ts_tablet_manager.cc:1120] Deleting tablet 25406a6f7c104671ac8b11de0c93b161
0503 02:12:19.488168 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.490488 (+ 2320us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x616000160354. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139eb8ba00 00007f139fe65b83 00007f13a28f957c 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74 00007f139336719b 00007f1393367018
0503 02:12:19.565706 (+ 75218us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.590787 (+ 25081us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.592987 (+ 2200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3573120,"tablet-delete.queue_time_us":915205}
I20240503 02:12:19.606262 5192 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:19.605288 5114 tablet_replica.cc:330] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:19.609299 5114 raft_consensus.cc:2229] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.610728 5114 raft_consensus.cc:2258] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.616561 5159 ts_tablet_manager.cc:1888] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:19.617708 5159 log.cc:1196] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c46ea1084304449e895be230d86471a9
I20240503 02:12:19.617658 5192 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.617971 5114 ts_tablet_manager.cc:1875] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.617355 5219 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:19.618896 5159 ts_tablet_manager.cc:1909] T c46ea1084304449e895be230d86471a9 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:19.622330 5177 consensus_queue.cc:1035] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:19.626549 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9db92b88674d908b1dea62e581caac (DEFAULT_TABLE table=default.Table_8fb97a4f46484d98a7e0d2d19788c5fb [id=608b3c3d30b14ddca9b21dcf3f985e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.627487 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9db92b88674d908b1dea62e581caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.628129 5176 mvcc.cc:205] Tried to move back new op lower bound from 7023420781192228864 to 7023420779532800000. Current Snapshot: MvccSnapshot[applied={T|T < 7023420781192228864}]
I20240503 02:12:19.628695 5159 tablet_replica.cc:330] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:19.628888 5159 raft_consensus.cc:2229] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.629101 5159 raft_consensus.cc:2258] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.626680 5219 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.630394 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:19.635253 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9db92b88674d908b1dea62e581caac (DEFAULT_TABLE table=default.Table_8fb97a4f46484d98a7e0d2d19788c5fb [id=608b3c3d30b14ddca9b21dcf3f985e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.636673 5103 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.637146 5192 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240503 02:12:19.638036 5159 ts_tablet_manager.cc:1875] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.637197 5103 ts_tablet_manager.cc:1367] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.154s user 0.007s sys 0.000s
I20240503 02:12:19.639607 5103 raft_consensus.cc:348] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.640030 5103 raft_consensus.cc:374] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.640167 5103 raft_consensus.cc:729] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.640425 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9db92b88674d908b1dea62e581caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.640626 5103 consensus_queue.cc:260] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.643131 5103 ts_tablet_manager.cc:1398] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:12:19.643671 5103 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.645812 4462 leader_election.cc:302] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:19.646183 4988 raft_consensus.cc:2726] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:19.646458 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 301318us
I20240503 02:12:19.647397 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ddc725b90349e38301cfc29c21d848" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:19.647687 4529 raft_consensus.cc:2379] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:19.648907 5115 ts_tablet_manager.cc:1875] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.651746 5103 tablet_bootstrap.cc:654] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.653847 4495 tablet_service.cc:1467] Processing CreateTablet for tablet f40706400f114def9e14e6d1d7114962 (DEFAULT_TABLE table=default.Table_cced2ab4f3d94350970ba6daf24e51b4 [id=76d119b7d82c4aaba94801c863535173]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.654610 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40706400f114def9e14e6d1d7114962. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.643711 5114 ts_tablet_manager.cc:1888] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.655573 5114 log.cc:1196] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/88f54d953fb24a3ea4bb21b6cf395edf
I20240503 02:12:19.656127 5219 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240503 02:12:19.656350 5114 ts_tablet_manager.cc:1909] T 88f54d953fb24a3ea4bb21b6cf395edf P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:19.657347 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 58) took 1010 ms. Trace:
I20240503 02:12:19.657521 5114 rpcz_store.cc:276] 0503 02:12:18.646898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.648997 (+ 2099us) service_pool.cc:225] Handling call
0503 02:12:19.604698 (+955701us) ts_tablet_manager.cc:1120] Deleting tablet 88f54d953fb24a3ea4bb21b6cf395edf
0503 02:12:19.604708 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.640471 (+ 35763us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.643580 (+ 3109us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.657285 (+ 13705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955272}
I20240503 02:12:19.659891 5114 tablet_replica.cc:330] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:19.660128 5114 raft_consensus.cc:2229] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.660329 5114 raft_consensus.cc:2258] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.662475 5114 ts_tablet_manager.cc:1875] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.662811 4782 tablet_service.cc:1467] Processing CreateTablet for tablet f40706400f114def9e14e6d1d7114962 (DEFAULT_TABLE table=default.Table_cced2ab4f3d94350970ba6daf24e51b4 [id=76d119b7d82c4aaba94801c863535173]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.663889 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40706400f114def9e14e6d1d7114962. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.668777 5106 tablet_bootstrap.cc:654] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.671171 5103 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.671372 5103 ts_tablet_manager.cc:1367] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240503 02:12:19.672317 5105 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:19.672490 5103 raft_consensus.cc:348] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.672791 5103 raft_consensus.cc:374] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.672904 5103 raft_consensus.cc:729] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.673231 5103 consensus_queue.cc:260] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.675268 5103 ts_tablet_manager.cc:1398] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:19.675597 5103 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.678972 5105 ts_tablet_manager.cc:1367] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.109s user 0.005s sys 0.000s
I20240503 02:12:19.679992 5105 raft_consensus.cc:348] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.680172 5103 tablet_bootstrap.cc:654] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.680349 5105 raft_consensus.cc:374] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.680500 5105 raft_consensus.cc:729] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.680815 5105 consensus_queue.cc:260] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.696377 4500 tablet_service.cc:1467] Processing CreateTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 (DEFAULT_TABLE table=default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.696851 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdefb33fe6e4a9893dd66f6252395d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.697752 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (DEFAULT_TABLE table=default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.698344 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.701300 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 959d998ec4ec4a52a6031650f2da4691 (DEFAULT_TABLE table=default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.701776 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959d998ec4ec4a52a6031650f2da4691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.704811 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7d1210a1217247b9801226055e2f2195
I20240503 02:12:19.580582 5216 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.696780 4784 tablet_service.cc:1467] Processing CreateTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 (DEFAULT_TABLE table=default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.706702 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdefb33fe6e4a9893dd66f6252395d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.697219 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 959d998ec4ec4a52a6031650f2da4691 (DEFAULT_TABLE table=default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.707304 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959d998ec4ec4a52a6031650f2da4691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.697595 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (DEFAULT_TABLE table=default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.714429 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.716123 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.716977 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.723878 4778 tablet_service.cc:1467] Processing CreateTablet for tablet dedead5a2bf347f6991cce8e9b4db708 (DEFAULT_TABLE table=default.Table_6ee3f7d7e571413dba0fe216678bf70d [id=01ac062e9b344819ad99b55ae572916d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.724377 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedead5a2bf347f6991cce8e9b4db708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.729458 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 546afc74e6c54f8aa5f7c684484d610b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.730067 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9db92b88674d908b1dea62e581caac (DEFAULT_TABLE table=default.Table_8fb97a4f46484d98a7e0d2d19788c5fb [id=608b3c3d30b14ddca9b21dcf3f985e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.730479 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9db92b88674d908b1dea62e581caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.731411 4784 tablet_service.cc:1467] Processing CreateTablet for tablet b28bb9ac773c4bc685cb8193e574460b (DEFAULT_TABLE table=default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.731846 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bb9ac773c4bc685cb8193e574460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.734463 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.735278 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fd4fb88436498999b5f339052ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:19.740270 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_eb6d6e34a8184594bd1d184b2e82177c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.721382 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a258c9bad19a4ce393deddba14dfcdbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.741765 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb6d6e34a8184594bd1d184b2e82177c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.742165 4650 tablet_service.cc:1467] Processing CreateTablet for tablet f40706400f114def9e14e6d1d7114962 (DEFAULT_TABLE table=default.Table_cced2ab4f3d94350970ba6daf24e51b4 [id=76d119b7d82c4aaba94801c863535173]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.742878 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40706400f114def9e14e6d1d7114962. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.748585 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2900c097d919482c9ee48d340935f0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.749258 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fe93cfdc45fd4972aa641dc189805745" } new_table_name: "default.Table_90d95bd149e04a83b065dba3bf38e43e" modify_external_catalogs: true
I20240503 02:12:19.750838 4645 tablet_service.cc:1467] Processing CreateTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 (DEFAULT_TABLE table=default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.753710 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 959d998ec4ec4a52a6031650f2da4691 (DEFAULT_TABLE table=default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.756822 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b7daa5af6c8044178b9f84aaba4cca33
I20240503 02:12:19.757460 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_828411f9b595429a8ca36e4b054c14f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:19.758037 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5c5ea1ab22594d6b81fbd5b536231007" } modify_external_catalogs: true
I20240503 02:12:19.759351 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (DEFAULT_TABLE table=default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.761652 4652 tablet_service.cc:1467] Processing CreateTablet for tablet b28bb9ac773c4bc685cb8193e574460b (DEFAULT_TABLE table=default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:19.763736 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a258c9bad19a4ce393deddba14dfcdbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:19.764659 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2900c097d919482c9ee48d340935f0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.766043 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:19.766768 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f663b1d9cbdd47cb82ea64536b0e5013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.768771 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f663b1d9cbdd47cb82ea64536b0e5013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.760859 5221 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.771289 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4f1cb580826c474cb8690af6403aefdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.771291 4608 leader_election.cc:341] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: fed654900a11471eaf6f7a198f8a985b
W20240503 02:12:19.773962 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f1cb580826c474cb8690af6403aefdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.777117 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5c5ea1ab22594d6b81fbd5b536231007" } modify_external_catalogs: true
I20240503 02:12:19.776778 4648 tablet_service.cc:1467] Processing CreateTablet for tablet dedead5a2bf347f6991cce8e9b4db708 (DEFAULT_TABLE table=default.Table_6ee3f7d7e571413dba0fe216678bf70d [id=01ac062e9b344819ad99b55ae572916d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.781801 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_8278654b5129427a9721eed448fc1758 [id=8b3bd9d6487e473089474aa5cc736cf4]). Expected version 7 got 6
I20240503 02:12:19.769555 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdefb33fe6e4a9893dd66f6252395d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.784417 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_828411f9b595429a8ca36e4b054c14f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.785480 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959d998ec4ec4a52a6031650f2da4691. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.787420 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4064ff59de474e0db3a578eae10c3fa6 consensus state has no leader
I20240503 02:12:19.792238 5106 tablet_bootstrap.cc:492] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:19.794550 5106 ts_tablet_manager.cc:1367] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.325s user 0.012s sys 0.005s
I20240503 02:12:19.806661 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2fd053276a0848d6bdf3010494bcaa0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.808490 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fd053276a0848d6bdf3010494bcaa0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.810112 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a95361cb35f04f1fb5b9e5ff04da58a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.811558 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a95361cb35f04f1fb5b9e5ff04da58a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.817662 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 3 got 2
I20240503 02:12:19.818091 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_8a018bc2b9e84350b29ef6447a8bb965 [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 3 got 2
I20240503 02:12:19.819501 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a15f08dd5c104b5295a3ebf6cd420b19" } new_table_name: "default.Table_8ed808d35bcc41bab738b11c2deca02c" modify_external_catalogs: true
I20240503 02:12:19.815704 5106 raft_consensus.cc:348] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.824854 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0799f0ea89c14b95b8593cd2e1423759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.825493 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0799f0ea89c14b95b8593cd2e1423759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.829064 5229 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:19.829597 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eb11fbb6817d4e0eb2e1ba3a72d8d920" } new_table_name: "default.Table_4ab6febbc9724cf78cc642abf953ee61" modify_external_catalogs: true
I20240503 02:12:19.829722 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.830430 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bb9ac773c4bc685cb8193e574460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.836280 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eb6d6e34a8184594bd1d184b2e82177c" } new_table_name: "default.Table_05e6a7077cf54c3ebcfb08151b563cdc" modify_external_catalogs: true
W20240503 02:12:19.829787 4608 leader_election.cc:341] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:19.849404 5106 raft_consensus.cc:374] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.849529 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f396f63d98b5441d8f9ccc43949c520e" } new_table_name: "default.Table_dde3e4c146264ea39a885cb77419db45" modify_external_catalogs: true
I20240503 02:12:19.849704 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedead5a2bf347f6991cce8e9b4db708. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:19.856204 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 73877.2us
I20240503 02:12:19.857604 5159 ts_tablet_manager.cc:1888] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.860325 5159 log.cc:1196] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/effb2ce36a31477382865461d89917be
I20240503 02:12:19.871064 5115 ts_tablet_manager.cc:1888] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.875157 5159 ts_tablet_manager.cc:1909] T effb2ce36a31477382865461d89917be P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:19.876574 4608 leader_election.cc:341] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b7ddc725b90349e38301cfc29c21d848
I20240503 02:12:19.876806 4608 leader_election.cc:302] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:19.877522 5200 raft_consensus.cc:3037] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.878273 5106 raft_consensus.cc:729] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.880816 5106 consensus_queue.cc:260] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.881630 5106 ts_tablet_manager.cc:1398] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.068s user 0.001s sys 0.002s
I20240503 02:12:19.882081 5106 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:19.883005 4608 leader_election.cc:341] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:19.883219 4608 leader_election.cc:302] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:19.883520 5208 raft_consensus.cc:2726] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.884860 5115 log.cc:1196] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/f22a4b8015354db493652d8f336a1265
I20240503 02:12:19.885445 5115 ts_tablet_manager.cc:1909] T f22a4b8015354db493652d8f336a1265 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:19.885857 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 32) took 2112 ms. Trace:
I20240503 02:12:19.886018 5115 rpcz_store.cc:276] 0503 02:12:17.773208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.773301 (+ 93us) service_pool.cc:225] Handling call
0503 02:12:19.543978 (+1770677us) ts_tablet_manager.cc:1120] Deleting tablet f22a4b8015354db493652d8f336a1265
0503 02:12:19.544018 (+ 40us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.763779 (+219761us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.870934 (+107155us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.885802 (+ 14868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759325}
I20240503 02:12:19.886277 5195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:19.886664 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 29) took 2144 ms. Trace:
I20240503 02:12:19.886763 5115 rpcz_store.cc:276] 0503 02:12:17.742554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.742909 (+ 355us) service_pool.cc:225] Handling call
0503 02:12:19.886485 (+2143576us) ts_tablet_manager.cc:1120] Deleting tablet e0fee5bceac941cd83471e5cad5dd463
0503 02:12:19.886494 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.886610 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062771}
W20240503 02:12:19.887089 4999 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.021s sys 0.052s
W20240503 02:12:19.887199 4999 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.021s sys 0.052s
I20240503 02:12:19.887293 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 31) took 2142 ms. Trace:
I20240503 02:12:19.887403 5115 rpcz_store.cc:276] 0503 02:12:17.748034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.748938 (+ 904us) service_pool.cc:225] Handling call
0503 02:12:19.887010 (+2138072us) ts_tablet_manager.cc:1120] Deleting tablet c46ea1084304449e895be230d86471a9
0503 02:12:19.887020 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.887112 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013852}
I20240503 02:12:19.887794 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 35) took 1644 ms. Trace:
I20240503 02:12:19.887888 5115 rpcz_store.cc:276] 0503 02:12:18.243603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.243847 (+ 244us) service_pool.cc:225] Handling call
0503 02:12:19.887649 (+1643802us) ts_tablet_manager.cc:1120] Deleting tablet a7fb095705c641ce929d9cc5029de3b9
0503 02:12:19.887659 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.887734 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643387}
W20240503 02:12:19.888006 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a829c48a5c24d6d9e6329ac34151d1f consensus state has no leader
I20240503 02:12:19.888268 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 36) took 1642 ms. Trace:
I20240503 02:12:19.888338 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet a295609170a8482f8373dbf817b1eea1 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.888403 5115 rpcz_store.cc:276] 0503 02:12:18.246210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.246540 (+ 330us) service_pool.cc:225] Handling call
0503 02:12:19.888115 (+1641575us) ts_tablet_manager.cc:1120] Deleting tablet effb2ce36a31477382865461d89917be
0503 02:12:19.888123 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.888216 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641290}
I20240503 02:12:19.888832 5221 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:19.886520 5195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:19.895185 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4967
I20240503 02:12:19.895726 5200 raft_consensus.cc:2726] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:19.898787 5219 tablet.cc:1717] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:19.899133 5229 tablet.cc:1717] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:19.899207 5220 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:19.909068 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet ced251fecca84866ab3ff15bd225e2b7 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:19.909909 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8750b02c5936412db525d2d3e77befeb has no consensus state
I20240503 02:12:19.909987 5229 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:19.911509 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet ced251fecca84866ab3ff15bd225e2b7 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.915133 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 634beeb9d8394b1da8c4ad79ba7e26ac from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.917207 4506 tablet_service.cc:1467] Processing CreateTablet for tablet b28bb9ac773c4bc685cb8193e574460b (DEFAULT_TABLE table=default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.917691 5106 tablet_bootstrap.cc:654] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.919356 4782 tablet_service.cc:1467] Processing CreateTablet for tablet c0c18cecefab4b6caed0039d09cc1890 (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.919901 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c18cecefab4b6caed0039d09cc1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.921622 5115 tablet_replica.cc:330] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:19.922099 5115 raft_consensus.cc:2229] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.922569 5115 raft_consensus.cc:2258] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.929811 5229 tablet.cc:1722] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240503 02:12:19.929824 5115 ts_tablet_manager.cc:1875] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.933017 4490 tablet_service.cc:1467] Processing CreateTablet for tablet dedead5a2bf347f6991cce8e9b4db708 (DEFAULT_TABLE table=default.Table_6ee3f7d7e571413dba0fe216678bf70d [id=01ac062e9b344819ad99b55ae572916d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:19.935137 5231 tablet.cc:1717] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:19.937469 5114 ts_tablet_manager.cc:1888] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.937659 5114 log.cc:1196] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/110c67212ec345028024da32968869ba
I20240503 02:12:19.938067 5114 ts_tablet_manager.cc:1909] T 110c67212ec345028024da32968869ba P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:19.942037 4988 raft_consensus.cc:482] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:19.942546 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bb9ac773c4bc685cb8193e574460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.943872 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedead5a2bf347f6991cce8e9b4db708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.945084 4988 raft_consensus.cc:504] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.945833 4988 leader_election.cc:288] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:19.947923 5192 tablet.cc:1717] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:19.949056 5115 ts_tablet_manager.cc:1888] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:19.949194 5115 log.cc:1196] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/fed654900a11471eaf6f7a198f8a985b
I20240503 02:12:19.952148 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c126b8ef15c5432abf037e28a5eeba58" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:19.952476 5106 tablet_bootstrap.cc:492] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:19.952729 5106 ts_tablet_manager.cc:1367] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.002s
W20240503 02:12:19.952780 4462 leader_election.cc:341] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:19.953670 5106 raft_consensus.cc:348] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.961023 5115 ts_tablet_manager.cc:1909] T fed654900a11471eaf6f7a198f8a985b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:19.961041 5106 raft_consensus.cc:374] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.961211 5106 raft_consensus.cc:729] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.961542 5106 consensus_queue.cc:260] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.962291 5106 ts_tablet_manager.cc:1398] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:12:19.962684 5106 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:19.963807 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 37) took 1709 ms. Trace:
I20240503 02:12:19.963955 5115 rpcz_store.cc:276] 0503 02:12:18.254355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.254415 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:19.921530 (+1667115us) ts_tablet_manager.cc:1120] Deleting tablet fed654900a11471eaf6f7a198f8a985b
0503 02:12:19.921539 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.934615 (+ 13076us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.948868 (+ 14253us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.963764 (+ 14896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666836}
I20240503 02:12:19.964999 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 64) took 1266 ms. Trace:
I20240503 02:12:19.965435 5114 rpcz_store.cc:276] 0503 02:12:18.698333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.698382 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:19.659819 (+961437us) ts_tablet_manager.cc:1120] Deleting tablet 110c67212ec345028024da32968869ba
0503 02:12:19.659827 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.667198 (+ 7371us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.730970 (+ 63772us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.964904 (+233934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961200}
I20240503 02:12:19.965842 5114 tablet_replica.cc:330] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:19.964545 5106 tablet_bootstrap.cc:654] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.966051 5114 raft_consensus.cc:2229] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.966385 5115 tablet_replica.cc:330] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:19.966368 5114 raft_consensus.cc:2258] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.967036 5114 ts_tablet_manager.cc:1875] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.971621 5115 raft_consensus.cc:2229] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.977161 5115 raft_consensus.cc:2258] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.978667 5115 ts_tablet_manager.cc:1875] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:19.985985 5106 tablet_bootstrap.cc:492] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:19.986162 5106 ts_tablet_manager.cc:1367] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240503 02:12:19.986768 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c126b8ef15c5432abf037e28a5eeba58" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:19.987234 4810 raft_consensus.cc:2454] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:19.987322 5106 raft_consensus.cc:348] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.987599 5106 raft_consensus.cc:374] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.987687 5106 raft_consensus.cc:729] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:19.987799 4462 leader_election.cc:302] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:19.987948 5106 consensus_queue.cc:260] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:19.988569 5106 ts_tablet_manager.cc:1398] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:19.988725 4988 raft_consensus.cc:2781] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:19.988840 4988 raft_consensus.cc:482] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:19.989192 4988 raft_consensus.cc:3037] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:19.990429 5103 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:19.990690 5103 ts_tablet_manager.cc:1367] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.315s user 0.013s sys 0.017s
I20240503 02:12:19.991559 5103 raft_consensus.cc:348] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.991887 5103 raft_consensus.cc:374] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:19.992002 5103 raft_consensus.cc:729] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:19.992491 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a295609170a8482f8373dbf817b1eea1 consensus state has no leader
I20240503 02:12:19.992321 5103 consensus_queue.cc:260] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:19.992818 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 22eec2c9a5654a4a8d6c3011ebc3c67c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:19.993358 5106 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:19.993897 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be08a6b601e14feb9ab4d23eb62484a6" } modify_external_catalogs: true
I20240503 02:12:19.994122 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be08a6b601e14feb9ab4d23eb62484a6" } modify_external_catalogs: true
I20240503 02:12:19.994314 5103 ts_tablet_manager.cc:1398] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:19.994712 5103 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:19.995247 5106 tablet_bootstrap.cc:654] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:19.998303 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8278654b5129427a9721eed448fc1758" } modify_external_catalogs: true
I20240503 02:12:19.998528 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8278654b5129427a9721eed448fc1758" } modify_external_catalogs: true
I20240503 02:12:19.999233 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1f7ba66d347245859fec5ef1be88f98f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:19.999852 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f7ba66d347245859fec5ef1be88f98f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:19.992326 4988 raft_consensus.cc:504] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.001727 4988 leader_election.cc:288] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.002583 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c126b8ef15c5432abf037e28a5eeba58" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:20.003784 5105 ts_tablet_manager.cc:1398] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.325s user 0.002s sys 0.000s
W20240503 02:12:20.003919 4462 leader_election.cc:341] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.004967 5103 tablet_bootstrap.cc:654] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.007495 5105 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.008486 5219 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.008847 5192 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.011539 5115 ts_tablet_manager.cc:1888] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.014673 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 37a54873b48847c39a2b961ab5c3806c (DEFAULT_TABLE table=default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.015158 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a54873b48847c39a2b961ab5c3806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.015095 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 79884b3e73904c708d0d48bca85351b7 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.015444 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79884b3e73904c708d0d48bca85351b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.021209 5103 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.021415 5103 ts_tablet_manager.cc:1367] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240503 02:12:20.021625 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.022202 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.022271 5103 raft_consensus.cc:348] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.022756 5103 raft_consensus.cc:374] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.022905 5103 raft_consensus.cc:729] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.023231 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.023466 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.023486 4497 tablet_service.cc:1467] Processing CreateTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 (DEFAULT_TABLE table=default.Table_a15f08dd5c104b5295a3ebf6cd420b19 [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.023450 5103 consensus_queue.cc:260] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.023846 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b14241dd76864e8a93d0279d3ab8f3ac (DEFAULT_TABLE table=default.Table_3d4325dad27b49bfa8a8e38c55372f6c [id=713930e2953343799de0e9c465e1c083]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.023936 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05d88a5c49b43bea6d46feccaaa6920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.024219 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14241dd76864e8a93d0279d3ab8f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.024140 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 851756dc1a5942919c85967f83eca29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 851756dc1a5942919c85967f83eca29a already in progress: creating tablet
I20240503 02:12:20.024864 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c126b8ef15c5432abf037e28a5eeba58" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:20.025060 5103 ts_tablet_manager.cc:1398] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:20.025100 4810 raft_consensus.cc:3037] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.025389 5103 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.027194 5103 tablet_bootstrap.cc:654] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.028811 4810 raft_consensus.cc:2454] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:20.029104 4785 tablet_service.cc:1467] Processing CreateTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 (DEFAULT_TABLE table=default.Table_a15f08dd5c104b5295a3ebf6cd420b19 [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.029392 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 37a54873b48847c39a2b961ab5c3806c (DEFAULT_TABLE table=default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.029644 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05d88a5c49b43bea6d46feccaaa6920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.029807 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a54873b48847c39a2b961ab5c3806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.033463 4462 leader_election.cc:302] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:20.033823 4988 raft_consensus.cc:2781] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:20.034103 4988 raft_consensus.cc:686] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:20.034528 4988 consensus_queue.cc:237] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.035315 5103 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.035494 5103 ts_tablet_manager.cc:1367] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 02:12:20.035944 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.036388 4495 tablet_service.cc:1467] Processing CreateTablet for tablet b14241dd76864e8a93d0279d3ab8f3ac (DEFAULT_TABLE table=default.Table_3d4325dad27b49bfa8a8e38c55372f6c [id=713930e2953343799de0e9c465e1c083]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.036834 5103 raft_consensus.cc:348] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.037096 5103 raft_consensus.cc:374] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.037171 5103 raft_consensus.cc:729] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.037385 5103 consensus_queue.cc:260] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.037839 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.038003 5103 ts_tablet_manager.cc:1398] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:20.038321 5103 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.038612 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14241dd76864e8a93d0279d3ab8f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.040058 5105 tablet_bootstrap.cc:654] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.041294 5115 log.cc:1196] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:20.041715 5115 ts_tablet_manager.cc:1909] T 461a6b30ab59485ea1a2ef8ae9815e04 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:20.042779 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 38) took 1771 ms. Trace:
I20240503 02:12:20.042883 5115 rpcz_store.cc:276] 0503 02:12:18.271451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.271541 (+ 90us) service_pool.cc:225] Handling call
0503 02:12:19.966308 (+1694767us) ts_tablet_manager.cc:1120] Deleting tablet 461a6b30ab59485ea1a2ef8ae9815e04
0503 02:12:19.966318 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.983584 (+ 17266us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.011380 (+ 27796us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.042521 (+ 31141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694466}
I20240503 02:12:20.043192 5103 tablet_bootstrap.cc:654] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.043658 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 48) took 1517 ms. Trace:
I20240503 02:12:20.043759 5115 rpcz_store.cc:276] 0503 02:12:18.526148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.526390 (+ 242us) service_pool.cc:225] Handling call
0503 02:12:20.043208 (+1516818us) ts_tablet_manager.cc:1120] Deleting tablet 634beeb9d8394b1da8c4ad79ba7e26ac
0503 02:12:20.043223 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.043353 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516505}
I20240503 02:12:20.044096 5115 tablet_replica.cc:330] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:20.045053 5221 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.045554 5200 raft_consensus.cc:482] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.045698 5200 raft_consensus.cc:504] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.046422 5200 leader_election.cc:288] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.047317 4648 tablet_service.cc:1467] Processing CreateTablet for tablet c0c18cecefab4b6caed0039d09cc1890 (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.047744 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c18cecefab4b6caed0039d09cc1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.049226 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 79884b3e73904c708d0d48bca85351b7 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.049609 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79884b3e73904c708d0d48bca85351b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.052636 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 37a54873b48847c39a2b961ab5c3806c (DEFAULT_TABLE table=default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.053089 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a54873b48847c39a2b961ab5c3806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.047574 5216 tablet.cc:1717] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:20.057067 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:20.057519 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2a51563396614c54b980f0679e84a2b1 because the tablet was not found. No further retry: Not found: Tablet not found: 2a51563396614c54b980f0679e84a2b1
W20240503 02:12:20.057803 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ab3c55f80a644cefa87b5769f15bb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab3c55f80a644cefa87b5769f15bb7f0 already in progress: creating tablet
W20240503 02:12:20.057966 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b487131ac3a842009b3fe54578ea091d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b487131ac3a842009b3fe54578ea091d already in progress: creating tablet
W20240503 02:12:20.058413 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fc887bc4db1c4e268c4c9f23096d704c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc887bc4db1c4e268c4c9f23096d704c already in progress: creating tablet
W20240503 02:12:20.058630 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ced251fecca84866ab3ff15bd225e2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced251fecca84866ab3ff15bd225e2b7 already in progress: creating tablet
I20240503 02:12:20.058807 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 88f54d953fb24a3ea4bb21b6cf395edf (table default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]) successfully deleted
I20240503 02:12:20.059064 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.059924 4650 tablet_service.cc:1467] Processing CreateTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 (DEFAULT_TABLE table=default.Table_a15f08dd5c104b5295a3ebf6cd420b19 [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.060324 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05d88a5c49b43bea6d46feccaaa6920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.062062 5115 raft_consensus.cc:2229] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.062597 5115 raft_consensus.cc:2258] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.063555 5115 ts_tablet_manager.cc:1875] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.070933 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.072522 5106 tablet_bootstrap.cc:492] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:20.072727 5106 ts_tablet_manager.cc:1367] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.005s
I20240503 02:12:20.073616 5106 raft_consensus.cc:348] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.073853 5106 raft_consensus.cc:374] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.073935 5106 raft_consensus.cc:729] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.074223 5106 consensus_queue.cc:260] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.075203 5115 ts_tablet_manager.cc:1888] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.079303 5105 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.079504 5105 ts_tablet_manager.cc:1367] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240503 02:12:20.079672 5115 log.cc:1196] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:20.075438 5106 ts_tablet_manager.cc:1398] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:20.080258 5115 ts_tablet_manager.cc:1909] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:20.080338 5106 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:20.080405 5105 raft_consensus.cc:348] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.080777 5105 raft_consensus.cc:374] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.077390 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b14241dd76864e8a93d0279d3ab8f3ac (DEFAULT_TABLE table=default.Table_3d4325dad27b49bfa8a8e38c55372f6c [id=713930e2953343799de0e9c465e1c083]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.081326 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 47) took 1557 ms. Trace:
I20240503 02:12:20.083127 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_621dd8b829b34a0eafce9326b41c3a25" } modify_external_catalogs: true
I20240503 02:12:20.083374 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_621dd8b829b34a0eafce9326b41c3a25" } modify_external_catalogs: true
I20240503 02:12:20.085441 4506 tablet_service.cc:1467] Processing CreateTablet for tablet c0c18cecefab4b6caed0039d09cc1890 (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.086256 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 25406a6f7c104671ac8b11de0c93b161 (table default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]) successfully deleted
I20240503 02:12:20.086670 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c18cecefab4b6caed0039d09cc1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.088488 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 79884b3e73904c708d0d48bca85351b7 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:20.092546 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e05d88a5c49b43bea6d46feccaaa6920 consensus state has no leader
I20240503 02:12:20.093124 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79884b3e73904c708d0d48bca85351b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.093029 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_489fd4b2341e48ee904ccad1bdad6d98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.093613 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_489fd4b2341e48ee904ccad1bdad6d98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.091899 4269 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_8278654b5129427a9721eed448fc1758 [id=8b3bd9d6487e473089474aa5cc736cf4] alter complete (version 7)
I20240503 02:12:20.095139 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ffd19b3ced6f4f58b7bc18123f9246c6" } new_table_name: "default.Table_27d1ed70e95c4b648e63e37a036f1b42" modify_external_catalogs: true
I20240503 02:12:20.098073 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fed654900a11471eaf6f7a198f8a985b (table default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]) successfully deleted
I20240503 02:12:20.098547 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c46ea1084304449e895be230d86471a9 (table default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]) successfully deleted
W20240503 02:12:20.098793 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c823155b774e42a5a422994903eae167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c823155b774e42a5a422994903eae167 already in progress: creating tablet
W20240503 02:12:20.099016 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet bf9c07c7cbb743d4a8e9610281a00f94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf9c07c7cbb743d4a8e9610281a00f94 already in progress: creating tablet
W20240503 02:12:20.099220 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 546afc74e6c54f8aa5f7c684484d610b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 546afc74e6c54f8aa5f7c684484d610b already in progress: creating tablet
I20240503 02:12:20.101008 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet bc0e75639a4f4f168f9d833b090ae76a
I20240503 02:12:20.102032 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e45d748d1bce468aac5ffaa88cf90f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.102769 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e45d748d1bce468aac5ffaa88cf90f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.110999 5105 raft_consensus.cc:729] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.111929 5105 consensus_queue.cc:260] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.114638 5105 ts_tablet_manager.cc:1398] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.035s user 0.000s sys 0.003s
I20240503 02:12:20.114622 5192 tablet.cc:1722] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:20.115648 5199 raft_consensus.cc:482] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.115828 5199 raft_consensus.cc:504] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.116942 5199 leader_election.cc:288] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.117998 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 387a596cfe654aa99e83d070b7884e6b
I20240503 02:12:20.118355 5105 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.120131 5105 tablet_bootstrap.cc:654] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.120177 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a295609170a8482f8373dbf817b1eea1
I20240503 02:12:20.120740 4273 catalog_manager.cc:6554] ReplaceTablet: tablet a295609170a8482f8373dbf817b1eea1 deleted and replaced by tablet 22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:20.122684 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_de35f554f3f64611a5a03c54e5c17e04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.123473 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de35f554f3f64611a5a03c54e5c17e04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.126447 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet effb2ce36a31477382865461d89917be (table default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]) successfully deleted
W20240503 02:12:20.126844 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 14fd4fb88436498999b5f339052ac0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fd4fb88436498999b5f339052ac0b6 already in progress: creating tablet
I20240503 02:12:20.127069 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 894b753b71a04193905a190da320a260
W20240503 02:12:20.127053 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fe5117d4d59b4f0b87c32e8b27b085d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5117d4d59b4f0b87c32e8b27b085d9 already in progress: creating tablet
W20240503 02:12:20.127516 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7d5e627d818245e0bdeb16531499f7d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d5e627d818245e0bdeb16531499f7d5 already in progress: creating tablet
W20240503 02:12:20.127702 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 27e8c98d5ea84179a91fdbea8657d6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27e8c98d5ea84179a91fdbea8657d6e6 already in progress: creating tablet
W20240503 02:12:20.127908 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet acdefb33fe6e4a9893dd66f6252395d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acdefb33fe6e4a9893dd66f6252395d7 already in progress: creating tablet
I20240503 02:12:20.127955 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_96c3eaec82a44cecbc28140867a68a73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.128525 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96c3eaec82a44cecbc28140867a68a73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.131924 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:20.132849 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_31aa999083a74c309bd7470f711571a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.128991 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet c126b8ef15c5432abf037e28a5eeba58 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d5e627d818245e0bdeb16531499f7d5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.136004 5159 tablet_replica.cc:330] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.136245 5159 raft_consensus.cc:2229] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.136406 5159 raft_consensus.cc:2258] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.136994 5159 ts_tablet_manager.cc:1875] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.138790 5114 ts_tablet_manager.cc:1888] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.138934 5114 log.cc:1196] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b7ddc725b90349e38301cfc29c21d848
I20240503 02:12:20.139451 5114 ts_tablet_manager.cc:1909] T b7ddc725b90349e38301cfc29c21d848 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.140295 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 65) took 1441 ms. Trace:
I20240503 02:12:20.140538 5114 rpcz_store.cc:276] 0503 02:12:18.698884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.698928 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:19.965758 (+1266830us) ts_tablet_manager.cc:1120] Deleting tablet b7ddc725b90349e38301cfc29c21d848
0503 02:12:19.965769 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:19.982408 (+ 16639us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.138587 (+156179us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.140243 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266652}
I20240503 02:12:20.144186 5115 rpcz_store.cc:276] 0503 02:12:18.524242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.524605 (+ 363us) service_pool.cc:225] Handling call
0503 02:12:20.043975 (+1519370us) ts_tablet_manager.cc:1120] Deleting tablet 25406a6f7c104671ac8b11de0c93b161
0503 02:12:20.043984 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.062445 (+ 18461us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x616000024d54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9974ba00 00007f1c9aa25b83 00007f1c9d4b957c 00007f1c9d4eb8d4 00007f1c9d4ccbc0 00007f1c9d4cc94d 0000000000a60bcc 00007f1c8df23b74 00007f1c8df2719b 00007f1c8df27018
0503 02:12:20.066937 (+ 4492us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.075037 (+ 8100us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.081231 (+ 6194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40817536,"tablet-delete.queue_time_us":1503612}
I20240503 02:12:20.145126 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 57) took 1455 ms. Trace:
I20240503 02:12:20.129201 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.146401 5115 rpcz_store.cc:276] 0503 02:12:18.689616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.689675 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:20.144874 (+1455199us) ts_tablet_manager.cc:1120] Deleting tablet 0f72aa1636df46ce9eec9c69c2f70c49
0503 02:12:20.144891 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.145076 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454850}
I20240503 02:12:20.147430 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 58) took 1450 ms. Trace:
I20240503 02:12:20.147924 5159 ts_tablet_manager.cc:1888] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.148119 5159 log.cc:1196] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c126b8ef15c5432abf037e28a5eeba58
I20240503 02:12:20.148519 5159 ts_tablet_manager.cc:1909] T c126b8ef15c5432abf037e28a5eeba58 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.148630 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14241dd76864e8a93d0279d3ab8f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.148664 5115 rpcz_store.cc:276] 0503 02:12:18.697404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.697471 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:20.146776 (+1449305us) ts_tablet_manager.cc:1120] Deleting tablet 21e625e02b764ff2b7279b35dd10b08c
0503 02:12:20.146789 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.147393 (+ 604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448897}
I20240503 02:12:20.149062 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 59) took 1450 ms. Trace:
I20240503 02:12:20.149402 5115 rpcz_store.cc:276] 0503 02:12:18.698493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.698973 (+ 480us) service_pool.cc:225] Handling call
0503 02:12:20.148894 (+1449921us) ts_tablet_manager.cc:1120] Deleting tablet 110c67212ec345028024da32968869ba
0503 02:12:20.148902 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.149025 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449169}
I20240503 02:12:20.149713 5115 tablet_replica.cc:330] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:20.150027 4988 raft_consensus.cc:482] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.150270 5114 tablet_replica.cc:330] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:20.149957 4608 leader_election.cc:341] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:20.150013 5115 raft_consensus.cc:2229] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.150805 5115 raft_consensus.cc:2258] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.151717 5115 ts_tablet_manager.cc:1875] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.151767 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:20.152711 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:20.153648 5103 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.153800 5103 ts_tablet_manager.cc:1367] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.008s
W20240503 02:12:20.154285 4462 leader_election.cc:341] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.154557 5106 tablet_bootstrap.cc:654] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:20.154542 4462 leader_election.cc:341] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.154778 5103 raft_consensus.cc:348] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.155059 5103 raft_consensus.cc:374] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.155181 5103 raft_consensus.cc:729] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.155503 5103 consensus_queue.cc:260] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.150267 4988 raft_consensus.cc:504] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:20.157819 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 68767.2us
W20240503 02:12:20.159200 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31aa999083a74c309bd7470f711571a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.160270 4988 leader_election.cc:288] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.160293 5114 raft_consensus.cc:2229] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.160706 5192 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.164960 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.165057 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.165278 5114 raft_consensus.cc:2258] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.166050 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.166334 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.167778 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.167979 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 (DEFAULT_TABLE table=default.Table_c00f7cc799d34540b77a581d4a9a4561 [id=7118c326733f4c6695909797bd035f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.168438 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb69d5f4bd427a8e430a00fb076f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.168289 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_631045fb785640fca1c307babc5c2276"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.168627 5114 ts_tablet_manager.cc:1875] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:20.168994 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_631045fb785640fca1c307babc5c2276 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.170740 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.172493 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.174086 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.174417 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 (DEFAULT_TABLE table=default.Table_c00f7cc799d34540b77a581d4a9a4561 [id=7118c326733f4c6695909797bd035f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.154676 4462 leader_election.cc:302] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:20.175340 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 (DEFAULT_TABLE table=default.Table_c00f7cc799d34540b77a581d4a9a4561 [id=7118c326733f4c6695909797bd035f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.175796 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb69d5f4bd427a8e430a00fb076f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.176659 4988 raft_consensus.cc:2726] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:20.177925 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb69d5f4bd427a8e430a00fb076f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.181694 5200 raft_consensus.cc:482] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.181882 5200 raft_consensus.cc:504] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.182595 5200 leader_election.cc:288] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.182619 5115 ts_tablet_manager.cc:1888] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.183038 5115 log.cc:1196] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b7ddc725b90349e38301cfc29c21d848
I20240503 02:12:20.183406 5221 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.183662 5115 ts_tablet_manager.cc:1909] T b7ddc725b90349e38301cfc29c21d848 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:20.184134 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69392fa5f441049437639e143a7491" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:20.185129 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 60) took 1485 ms. Trace:
I20240503 02:12:20.185307 5115 rpcz_store.cc:276] 0503 02:12:18.699427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.700889 (+ 1462us) service_pool.cc:225] Handling call
0503 02:12:20.149651 (+1448762us) ts_tablet_manager.cc:1120] Deleting tablet b7ddc725b90349e38301cfc29c21d848
0503 02:12:20.149658 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.157854 (+ 8196us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.182400 (+ 24546us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.185064 (+ 2664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448461}
I20240503 02:12:20.185935 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 61) took 1485 ms. Trace:
I20240503 02:12:20.186111 5115 rpcz_store.cc:276] 0503 02:12:18.700171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.701802 (+ 1631us) service_pool.cc:225] Handling call
0503 02:12:20.185754 (+1483952us) ts_tablet_manager.cc:1120] Deleting tablet 7b69392fa5f441049437639e143a7491
0503 02:12:20.185763 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.185870 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483700}
I20240503 02:12:20.186601 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 67) took 1388 ms. Trace:
W20240503 02:12:20.186803 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 617ad14927c1441b800a204aefc2e619 has no consensus state
I20240503 02:12:20.186748 5115 rpcz_store.cc:276] 0503 02:12:18.798524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.798629 (+ 105us) service_pool.cc:225] Handling call
0503 02:12:20.186440 (+1387811us) ts_tablet_manager.cc:1120] Deleting tablet 003c9c3e0a1d400c94bfaefada2e9faa
0503 02:12:20.186449 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.186539 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1375069}
I20240503 02:12:20.187103 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2900c097d919482c9ee48d340935f0c3" } new_table_name: "default.Table_5ac4a60d6df343668e711966d75f932c" modify_external_catalogs: true
I20240503 02:12:20.187350 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 68) took 1380 ms. Trace:
I20240503 02:12:20.187297 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4e4a5e92b87343dc9067868e6ba38292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.187498 5115 rpcz_store.cc:276] 0503 02:12:18.806473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.806664 (+ 191us) service_pool.cc:225] Handling call
0503 02:12:20.187127 (+1380463us) ts_tablet_manager.cc:1120] Deleting tablet 003c9c3e0a1d400c94bfaefada2e9faa
0503 02:12:20.187134 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.187229 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373169}
W20240503 02:12:20.187940 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e4a5e92b87343dc9067868e6ba38292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.188098 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 62) took 1469 ms. Trace:
I20240503 02:12:20.188236 5115 rpcz_store.cc:276] 0503 02:12:18.718982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.721686 (+ 2704us) service_pool.cc:225] Handling call
0503 02:12:20.187901 (+1466215us) ts_tablet_manager.cc:1120] Deleting tablet c126b8ef15c5432abf037e28a5eeba58
0503 02:12:20.187911 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.187994 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341078}
I20240503 02:12:20.188632 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 48135062d92949dc8386c43cf103ced4
I20240503 02:12:20.188675 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 81) took 1102 ms. Trace:
I20240503 02:12:20.188866 5115 rpcz_store.cc:276] 0503 02:12:19.086427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.086949 (+ 522us) service_pool.cc:225] Handling call
0503 02:12:20.188524 (+1101575us) ts_tablet_manager.cc:1120] Deleting tablet ab3c55f80a644cefa87b5769f15bb7f0
0503 02:12:20.188533 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.188624 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089061}
I20240503 02:12:20.189241 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6ee3f7d7e571413dba0fe216678bf70d" } new_table_name: "default.Table_b405182d426d419d9ba68b4159595b78" modify_external_catalogs: true
I20240503 02:12:20.189332 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 83) took 1102 ms. Trace:
I20240503 02:12:20.189409 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_25ca6a7de9b2413ba4623e94f133fac8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.189529 5115 rpcz_store.cc:276] 0503 02:12:19.086740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.086849 (+ 109us) service_pool.cc:225] Handling call
0503 02:12:20.189200 (+1102351us) ts_tablet_manager.cc:1120] Deleting tablet 2a51563396614c54b980f0679e84a2b1
0503 02:12:20.189207 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.189281 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":918895}
W20240503 02:12:20.189965 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25ca6a7de9b2413ba4623e94f133fac8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.190110 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 82) took 1103 ms. Trace:
I20240503 02:12:20.190237 5115 rpcz_store.cc:276] 0503 02:12:19.086515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.086879 (+ 364us) service_pool.cc:225] Handling call
0503 02:12:20.189898 (+1103019us) ts_tablet_manager.cc:1120] Deleting tablet 851756dc1a5942919c85967f83eca29a
0503 02:12:20.189906 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.189992 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":918635}
I20240503 02:12:20.190775 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 80) took 1104 ms. Trace:
I20240503 02:12:20.190943 5115 rpcz_store.cc:276] 0503 02:12:19.086322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.086981 (+ 659us) service_pool.cc:225] Handling call
0503 02:12:20.190635 (+1103654us) ts_tablet_manager.cc:1120] Deleting tablet b487131ac3a842009b3fe54578ea091d
0503 02:12:20.190644 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.190728 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":990,"tablet-delete.queue_time_us":919373}
I20240503 02:12:20.191529 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 87) took 1092 ms. Trace:
I20240503 02:12:20.191680 5115 rpcz_store.cc:276] 0503 02:12:19.098851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.098914 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:20.191231 (+1092317us) ts_tablet_manager.cc:1120] Deleting tablet fc887bc4db1c4e268c4c9f23096d704c
0503 02:12:20.191364 (+ 133us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.191473 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":915670}
I20240503 02:12:20.192137 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 88) took 1093 ms. Trace:
I20240503 02:12:20.192286 5115 rpcz_store.cc:276] 0503 02:12:19.099032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.099084 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:20.191968 (+1092884us) ts_tablet_manager.cc:1120] Deleting tablet ced251fecca84866ab3ff15bd225e2b7
0503 02:12:20.191978 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.192089 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":916228}
I20240503 02:12:20.193013 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69392fa5f441049437639e143a7491" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:20.193274 4810 raft_consensus.cc:2454] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
W20240503 02:12:20.197016 4462 leader_election.cc:341] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.197469 4462 leader_election.cc:302] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:20.198086 5220 tablet.cc:1722] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:20.200615 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_31aa999083a74c309bd7470f711571a2" } new_table_name: "default.Table_02838524663542819d76fcab9d549504" modify_external_catalogs: true
W20240503 02:12:20.203145 4462 raft_consensus.cc:2670] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:20.204959 5105 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:20.205087 4608 leader_election.cc:341] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 25406a6f7c104671ac8b11de0c93b161
I20240503 02:12:20.205303 5105 ts_tablet_manager.cc:1367] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.000s
I20240503 02:12:20.206279 5105 raft_consensus.cc:348] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.206600 5105 raft_consensus.cc:374] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.206794 5105 raft_consensus.cc:729] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.207145 5105 consensus_queue.cc:260] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.209002 5105 ts_tablet_manager.cc:1398] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:20.209367 5105 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.211298 4778 tablet_service.cc:1467] Processing CreateTablet for tablet bbab4368e73b4944b9853c456a4db7bd (DEFAULT_TABLE table=default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.211598 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.211761 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbab4368e73b4944b9853c456a4db7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.211992 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9327a9e5472b4eda854320c3a95f8e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.215209 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 37a54873b48847c39a2b961ab5c3806c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.219127 4785 tablet_service.cc:1467] Processing CreateTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb (DEFAULT_TABLE table=default.Table_b6ddecc9e022479595b1ce5c50323184 [id=e08a62f9895e4721b5524313acf78175]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.219352 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0e75639a4f4f168f9d833b090ae76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.219659 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.219825 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc (DEFAULT_TABLE table=default.Table_bba46dea649d4895bacf0532ffeb8a7f [id=5febc608abd84e7ba26288f2e155d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.220332 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 09005fb01c384122a7310c1eea12d3f0 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.221042 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet a295609170a8482f8373dbf817b1eea1 with delete_type TABLET_DATA_DELETED (replaced by tablet 22776c81d1a24d368abb68274cf32a0b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.221443 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69392fa5f441049437639e143a7491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.221705 5159 tablet_replica.cc:330] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.221896 5159 raft_consensus.cc:2229] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.222088 5159 raft_consensus.cc:2258] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.222642 5159 ts_tablet_manager.cc:1875] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.219688 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b080fc5ffc564420a0b7f2b7460ebbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.222981 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09005fb01c384122a7310c1eea12d3f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.223662 4259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.17.126:43321 (error 108) [suppressed 7 similar messages]
I20240503 02:12:20.225270 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17afd88bf6c040ad9e8d571eccd348dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.234338 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.234988 4495 tablet_service.cc:1467] Processing CreateTablet for tablet bbab4368e73b4944b9853c456a4db7bd (DEFAULT_TABLE table=default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.235508 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbab4368e73b4944b9853c456a4db7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.235996 5114 ts_tablet_manager.cc:1888] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.236734 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 22eec2c9a5654a4a8d6c3011ebc3c67c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.237309 5114 log.cc:1196] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7b69392fa5f441049437639e143a7491
I20240503 02:12:20.239316 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0e75639a4f4f168f9d833b090ae76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.239531 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc (DEFAULT_TABLE table=default.Table_bba46dea649d4895bacf0532ffeb8a7f [id=5febc608abd84e7ba26288f2e155d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.239955 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17afd88bf6c040ad9e8d571eccd348dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.241010 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b69392fa5f441049437639e143a7491 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.245165 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 09005fb01c384122a7310c1eea12d3f0 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.245667 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09005fb01c384122a7310c1eea12d3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.246618 5200 raft_consensus.cc:482] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.246790 5200 raft_consensus.cc:504] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.247550 5200 leader_election.cc:288] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.249626 5159 ts_tablet_manager.cc:1888] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.249761 5159 log.cc:1196] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7b69392fa5f441049437639e143a7491
I20240503 02:12:20.250118 5159 ts_tablet_manager.cc:1909] T 7b69392fa5f441049437639e143a7491 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:20.251402 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19fb0929bd234788b5313d96d2e4845c has no consensus state
W20240503 02:12:20.251842 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dedead5a2bf347f6991cce8e9b4db708 consensus state has no leader
I20240503 02:12:20.253839 4270 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae [id=4d9a163c0c564955a47a65f799a0018c] alter complete (version 3)
I20240503 02:12:20.254345 5105 tablet_bootstrap.cc:654] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:20.262387 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b41dce88919456aa846dbdcf5e2759f has no consensus state
I20240503 02:12:20.264053 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.264426 4497 tablet_service.cc:1467] Processing CreateTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb (DEFAULT_TABLE table=default.Table_b6ddecc9e022479595b1ce5c50323184 [id=e08a62f9895e4721b5524313acf78175]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.264669 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.265132 5114 ts_tablet_manager.cc:1909] T 7b69392fa5f441049437639e143a7491 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.265859 5105 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.265864 5106 tablet_bootstrap.cc:492] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:20.266021 5106 ts_tablet_manager.cc:1367] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.186s user 0.006s sys 0.002s
I20240503 02:12:20.266032 5105 ts_tablet_manager.cc:1367] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.000s
I20240503 02:12:20.266409 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet a295609170a8482f8373dbf817b1eea1 with delete_type TABLET_DATA_DELETED (replaced by tablet 22776c81d1a24d368abb68274cf32a0b) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.266597 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.266798 5106 raft_consensus.cc:348] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.266798 5105 raft_consensus.cc:348] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.266990 5103 ts_tablet_manager.cc:1398] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.084s user 0.002s sys 0.002s
I20240503 02:12:20.267022 5106 raft_consensus.cc:374] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.267030 5105 raft_consensus.cc:374] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.267115 5105 raft_consensus.cc:729] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.267115 5106 raft_consensus.cc:729] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.267329 5103 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.267349 5106 consensus_queue.cc:260] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.268800 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9327a9e5472b4eda854320c3a95f8e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.268677 5105 consensus_queue.cc:260] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.271279 5105 ts_tablet_manager.cc:1398] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:20.272382 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b080fc5ffc564420a0b7f2b7460ebbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.272432 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 66) took 1568 ms. Trace:
I20240503 02:12:20.272532 5114 rpcz_store.cc:276] 0503 02:12:18.700334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.700386 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:20.140856 (+1440470us) ts_tablet_manager.cc:1120] Deleting tablet 7b69392fa5f441049437639e143a7491
0503 02:12:20.140866 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.160536 (+ 19670us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x616000170854. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139eb8ba00 00007f139fe65b83 00007f13a28f957c 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74 00007f139336719b 00007f1393367018
0503 02:12:20.175828 (+ 15292us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.235774 (+ 59946us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.268674 (+ 32900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17493376,"tablet-delete.queue_time_us":1440233}
I20240503 02:12:20.273396 5106 ts_tablet_manager.cc:1398] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:12:20.273905 5106 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:20.273980 5114 tablet_replica.cc:330] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.274183 5114 raft_consensus.cc:2229] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:20.275790 5106 tablet_bootstrap.cc:654] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.276669 4608 leader_election.cc:302] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:20.276947 4608 raft_consensus.cc:2670] T 25406a6f7c104671ac8b11de0c93b161 P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:20.277592 5114 pending_rounds.cc:62] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
W20240503 02:12:20.277570 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 79634.2us
I20240503 02:12:20.278082 5105 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.278232 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_febbb9150c994770b47d48864b59d021"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.278838 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_febbb9150c994770b47d48864b59d021 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.279811 5105 tablet_bootstrap.cc:654] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.282373 5103 tablet_bootstrap.cc:654] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.283025 5114 pending_rounds.cc:68] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420784785133568 op_type: NO_OP noop_request { }
I20240503 02:12:20.283581 5114 raft_consensus.cc:2866] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:20.283715 5114 raft_consensus.cc:2258] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.285065 5114 ts_tablet_manager.cc:1875] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.285784 4270 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228] alter complete (version 4)
I20240503 02:12:20.297693 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b8a86d9e96be4890bad78b29e651918a" } new_table_name: "default.Table_487c2ed324fb43f0a376681dd950cbad" modify_external_catalogs: true
I20240503 02:12:20.308009 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 22eec2c9a5654a4a8d6c3011ebc3c67c
I20240503 02:12:20.308502 4272 catalog_manager.cc:6554] ReplaceTablet: tablet 22eec2c9a5654a4a8d6c3011ebc3c67c deleted and replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7
I20240503 02:12:20.311731 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eeccbe7767854540840b84af4d2820cf" } modify_external_catalogs: true
I20240503 02:12:20.312095 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eeccbe7767854540840b84af4d2820cf" } modify_external_catalogs: true
I20240503 02:12:20.322093 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 37a54873b48847c39a2b961ab5c3806c
I20240503 02:12:20.322496 5114 ts_tablet_manager.cc:1888] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:20.322679 5114 log.cc:1196] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c126b8ef15c5432abf037e28a5eeba58
I20240503 02:12:20.322692 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 37a54873b48847c39a2b961ab5c3806c deleted and replaced by tablet b3223b35d26d425582e61f70f17fbdc9
I20240503 02:12:20.325017 5114 ts_tablet_manager.cc:1909] T c126b8ef15c5432abf037e28a5eeba58 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.325881 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 67) took 1606 ms. Trace:
I20240503 02:12:20.325973 5114 rpcz_store.cc:276] 0503 02:12:18.719777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.719841 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:20.272835 (+1552994us) ts_tablet_manager.cc:1120] Deleting tablet c126b8ef15c5432abf037e28a5eeba58
0503 02:12:20.272846 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.303198 (+ 30352us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.322290 (+ 19092us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.325824 (+ 3534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1552749}
I20240503 02:12:20.326961 5103 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.327109 5103 ts_tablet_manager.cc:1367] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.005s
I20240503 02:12:20.327804 5103 raft_consensus.cc:348] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.328013 5103 raft_consensus.cc:374] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.328078 5103 raft_consensus.cc:729] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.328362 5103 consensus_queue.cc:260] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.329047 5103 ts_tablet_manager.cc:1398] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:20.332715 5103 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.337010 5103 tablet_bootstrap.cc:654] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.348421 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 74) took 1548 ms. Trace:
I20240503 02:12:20.348558 5114 rpcz_store.cc:276] 0503 02:12:18.796869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.797241 (+ 372us) service_pool.cc:225] Handling call
0503 02:12:20.345468 (+1548227us) ts_tablet_manager.cc:1120] Deleting tablet 003c9c3e0a1d400c94bfaefada2e9faa
0503 02:12:20.345486 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.345665 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547825}
I20240503 02:12:20.349383 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 73) took 1553 ms. Trace:
I20240503 02:12:20.349517 5114 rpcz_store.cc:276] 0503 02:12:18.796013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:18.797689 (+ 1676us) service_pool.cc:225] Handling call
0503 02:12:20.349162 (+1551473us) ts_tablet_manager.cc:1120] Deleting tablet 003c9c3e0a1d400c94bfaefada2e9faa
0503 02:12:20.349175 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.349316 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551243}
I20240503 02:12:20.349946 5114 tablet_replica.cc:330] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.350211 5114 raft_consensus.cc:2229] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.350423 5114 raft_consensus.cc:2258] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.351464 5114 ts_tablet_manager.cc:1875] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.355100 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b69392fa5f441049437639e143a7491 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.359757 5103 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.359937 5103 ts_tablet_manager.cc:1367] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.001s
I20240503 02:12:20.360782 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8925d918e3de4d60b2f13a0c0e06f5be" } modify_external_catalogs: true
I20240503 02:12:20.360826 5103 raft_consensus.cc:348] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.361050 5103 raft_consensus.cc:374] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.361037 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8925d918e3de4d60b2f13a0c0e06f5be" } modify_external_catalogs: true
I20240503 02:12:20.361171 5103 raft_consensus.cc:729] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.361510 5103 consensus_queue.cc:260] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.361887 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4ab6febbc9724cf78cc642abf953ee61" } modify_external_catalogs: true
I20240503 02:12:20.362087 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4ab6febbc9724cf78cc642abf953ee61" } modify_external_catalogs: true
I20240503 02:12:20.362504 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5db43ecab50c42d6bd7cea5b311d5f95" } modify_external_catalogs: true
I20240503 02:12:20.362682 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5db43ecab50c42d6bd7cea5b311d5f95" } modify_external_catalogs: true
I20240503 02:12:20.364934 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_dacc67e3ff0c49fc9dddc714ec081269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.365599 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dacc67e3ff0c49fc9dddc714ec081269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.366166 5103 ts_tablet_manager.cc:1398] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:20.373054 5103 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.378592 5103 tablet_bootstrap.cc:654] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.378963 4270 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_8a018bc2b9e84350b29ef6447a8bb965 [id=ecdb526541774f2cbecc22c8efc4c3dd] alter complete (version 3)
W20240503 02:12:20.379688 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 155991us
I20240503 02:12:20.374526 5114 ts_tablet_manager.cc:1888] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.380151 5114 log.cc:1196] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c46ea1084304449e895be230d86471a9
I20240503 02:12:20.380558 5114 ts_tablet_manager.cc:1909] T c46ea1084304449e895be230d86471a9 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.380797 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 88f54d953fb24a3ea4bb21b6cf395edf (table default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]) successfully deleted
I20240503 02:12:20.381479 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 85) took 1292 ms. Trace:
I20240503 02:12:20.381613 5114 rpcz_store.cc:276] 0503 02:12:19.089479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.090022 (+ 543us) service_pool.cc:225] Handling call
0503 02:12:20.349868 (+1259846us) ts_tablet_manager.cc:1120] Deleting tablet c46ea1084304449e895be230d86471a9
0503 02:12:20.349876 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.370839 (+ 20963us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.374380 (+ 3541us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.381441 (+ 7061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129435}
I20240503 02:12:20.382068 5199 raft_consensus.cc:482] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.382241 5199 raft_consensus.cc:504] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.382351 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet f22a4b8015354db493652d8f336a1265 (table default.Table_aecae221afdc4f5ebe0462800c22c19f [id=f392f75ae3754c04849992c1f19fc098]) successfully deleted
W20240503 02:12:20.382576 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet e0fee5bceac941cd83471e5cad5dd463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fee5bceac941cd83471e5cad5dd463 already in progress: creating tablet
W20240503 02:12:20.382798 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c46ea1084304449e895be230d86471a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c46ea1084304449e895be230d86471a9 already in progress: creating tablet
I20240503 02:12:20.382911 5199 leader_election.cc:288] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:20.383000 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a7fb095705c641ce929d9cc5029de3b9 because the tablet was not found. No further retry: Not found: Tablet not found: a7fb095705c641ce929d9cc5029de3b9
I20240503 02:12:20.383109 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_8a018bc2b9e84350b29ef6447a8bb965 [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 3 got 2
W20240503 02:12:20.383512 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet effb2ce36a31477382865461d89917be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet effb2ce36a31477382865461d89917be already in progress: creating tablet
I20240503 02:12:20.383648 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487131ac3a842009b3fe54578ea091d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:20.383742 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet fed654900a11471eaf6f7a198f8a985b (table default.Table_3e37722f22364e62b5e10db841f1b963 [id=eb5462eb8b4743ca8e5b2e9f33956a10]) successfully deleted
I20240503 02:12:20.383703 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487131ac3a842009b3fe54578ea091d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:20.383913 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 461a6b30ab59485ea1a2ef8ae9815e04 (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
W20240503 02:12:20.384317 4462 leader_election.cc:341] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:20.384507 4462 leader_election.cc:341] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.384567 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 25406a6f7c104671ac8b11de0c93b161 (table default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]) successfully deleted
I20240503 02:12:20.384624 4462 leader_election.cc:302] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:20.384946 5106 tablet_bootstrap.cc:492] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:20.385202 5106 ts_tablet_manager.cc:1367] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.002s
I20240503 02:12:20.385229 5199 raft_consensus.cc:2726] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:20.385547 5114 tablet_replica.cc:330] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.385798 5114 raft_consensus.cc:2229] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.386008 5114 raft_consensus.cc:2258] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.386332 5106 raft_consensus.cc:348] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.386649 5106 raft_consensus.cc:374] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.386662 5114 ts_tablet_manager.cc:1875] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.386755 5106 raft_consensus.cc:729] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.387131 5106 consensus_queue.cc:260] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.387758 5106 ts_tablet_manager.cc:1398] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:20.388068 5106 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:20.388942 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0f72aa1636df46ce9eec9c69c2f70c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f72aa1636df46ce9eec9c69c2f70c49 already in progress: creating tablet
W20240503 02:12:20.389212 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 21e625e02b764ff2b7279b35dd10b08c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e625e02b764ff2b7279b35dd10b08c already in progress: creating tablet
W20240503 02:12:20.389437 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 110c67212ec345028024da32968869ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110c67212ec345028024da32968869ba already in progress: creating tablet
W20240503 02:12:20.389811 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7b69392fa5f441049437639e143a7491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b69392fa5f441049437639e143a7491 already in progress: creating tablet
W20240503 02:12:20.390065 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 003c9c3e0a1d400c94bfaefada2e9faa already in progress: creating tablet
W20240503 02:12:20.390276 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 003c9c3e0a1d400c94bfaefada2e9faa already in progress: creating tablet
W20240503 02:12:20.390578 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ab3c55f80a644cefa87b5769f15bb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab3c55f80a644cefa87b5769f15bb7f0 already in progress: creating tablet
I20240503 02:12:20.395583 5114 ts_tablet_manager.cc:1888] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.395793 5114 log.cc:1196] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/461a6b30ab59485ea1a2ef8ae9815e04
I20240503 02:12:20.396175 5114 ts_tablet_manager.cc:1909] T 461a6b30ab59485ea1a2ef8ae9815e04 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.397266 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 84) took 1308 ms. Trace:
I20240503 02:12:20.397425 5114 rpcz_store.cc:276] 0503 02:12:19.089259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.090049 (+ 790us) service_pool.cc:225] Handling call
0503 02:12:20.385467 (+1295418us) ts_tablet_manager.cc:1120] Deleting tablet 461a6b30ab59485ea1a2ef8ae9815e04
0503 02:12:20.385484 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.391720 (+ 6236us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.395437 (+ 3717us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.397205 (+ 1768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163691}
I20240503 02:12:20.392191 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c0626018603746ea91e3a2d6cfc37ef8" } modify_external_catalogs: true
I20240503 02:12:20.397802 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c0626018603746ea91e3a2d6cfc37ef8" } modify_external_catalogs: true
W20240503 02:12:20.397905 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2a51563396614c54b980f0679e84a2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a51563396614c54b980f0679e84a2b1 already in progress: creating tablet
I20240503 02:12:20.392293 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.398145 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 851756dc1a5942919c85967f83eca29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 851756dc1a5942919c85967f83eca29a already in progress: creating tablet
W20240503 02:12:20.398393 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b487131ac3a842009b3fe54578ea091d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b487131ac3a842009b3fe54578ea091d already in progress: creating tablet
W20240503 02:12:20.398578 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:20.398700 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ced251fecca84866ab3ff15bd225e2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced251fecca84866ab3ff15bd225e2b7 already in progress: creating tablet
I20240503 02:12:20.398934 5114 tablet_replica.cc:330] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:20.398917 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c823155b774e42a5a422994903eae167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c823155b774e42a5a422994903eae167 already in progress: creating tablet
I20240503 02:12:20.398869 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_febbb9150c994770b47d48864b59d021" } new_table_name: "default.Table_bbf1abd1706f4e85a126f2c232a55a8b" modify_external_catalogs: true
I20240503 02:12:20.399108 5114 raft_consensus.cc:2229] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.399317 5114 raft_consensus.cc:2258] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:20.396837 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46560ac6d94943d2a8fb3b3904f2b5b6 consensus state has no leader
I20240503 02:12:20.400169 5114 ts_tablet_manager.cc:1875] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.400255 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_39060d5d029f4211a2c27b981453e636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.400744 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39060d5d029f4211a2c27b981453e636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.405121 4268 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7
I20240503 02:12:20.405709 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.406313 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:20.409224 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet fe5117d4d59b4f0b87c32e8b27b085d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5117d4d59b4f0b87c32e8b27b085d9 already in progress: creating tablet
I20240503 02:12:20.409760 5105 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:20.409862 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 546afc74e6c54f8aa5f7c684484d610b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 546afc74e6c54f8aa5f7c684484d610b already in progress: creating tablet
W20240503 02:12:20.410081 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet bf9c07c7cbb743d4a8e9610281a00f94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf9c07c7cbb743d4a8e9610281a00f94 already in progress: creating tablet
I20240503 02:12:20.410125 5105 ts_tablet_manager.cc:1367] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.138s user 0.034s sys 0.077s
W20240503 02:12:20.410265 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 14fd4fb88436498999b5f339052ac0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fd4fb88436498999b5f339052ac0b6 already in progress: creating tablet
I20240503 02:12:20.410766 5114 ts_tablet_manager.cc:1888] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:20.410849 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7d5e627d818245e0bdeb16531499f7d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d5e627d818245e0bdeb16531499f7d5 already in progress: creating tablet
I20240503 02:12:20.410936 5114 log.cc:1196] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b487131ac3a842009b3fe54578ea091d
W20240503 02:12:20.411068 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 27e8c98d5ea84179a91fdbea8657d6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27e8c98d5ea84179a91fdbea8657d6e6 already in progress: creating tablet
W20240503 02:12:20.411293 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet acdefb33fe6e4a9893dd66f6252395d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acdefb33fe6e4a9893dd66f6252395d7 already in progress: creating tablet
I20240503 02:12:20.411374 5114 ts_tablet_manager.cc:1909] T b487131ac3a842009b3fe54578ea091d P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:20.411541 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7d1210a1217247b9801226055e2f2195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1210a1217247b9801226055e2f2195 already in progress: creating tablet
W20240503 02:12:20.411775 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b7daa5af6c8044178b9f84aaba4cca33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7daa5af6c8044178b9f84aaba4cca33 already in progress: creating tablet
W20240503 02:12:20.411960 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9f57f3833a9240c88e0de1b486fae265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f57f3833a9240c88e0de1b486fae265 already in progress: creating tablet
I20240503 02:12:20.412192 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 88) took 1322 ms. Trace:
I20240503 02:12:20.412467 5200 raft_consensus.cc:482] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.412736 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 25406a6f7c104671ac8b11de0c93b161 (table default.Table_58a13c96da8f4ca089be3162a5b53abb [id=376ff08e2257444aac746c84adf975cf]) successfully deleted
I20240503 02:12:20.412688 5200 raft_consensus.cc:504] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.413019 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 88f54d953fb24a3ea4bb21b6cf395edf (table default.Table_72b4b8bc07a14f3cb17ddd8ce9ad8908 [id=59dde771aa9e41a090eb1ab8191a67ac]) successfully deleted
W20240503 02:12:20.413214 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0f72aa1636df46ce9eec9c69c2f70c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f72aa1636df46ce9eec9c69c2f70c49 already in progress: creating tablet
I20240503 02:12:20.413317 5200 leader_election.cc:288] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.412297 5114 rpcz_store.cc:276] 0503 02:12:19.089859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.089986 (+ 127us) service_pool.cc:225] Handling call
0503 02:12:20.398846 (+1308860us) ts_tablet_manager.cc:1120] Deleting tablet b487131ac3a842009b3fe54578ea091d
0503 02:12:20.398857 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.406968 (+ 8111us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.410587 (+ 3619us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.412151 (+ 1564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165452}
W20240503 02:12:20.413408 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 21e625e02b764ff2b7279b35dd10b08c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e625e02b764ff2b7279b35dd10b08c already in progress: creating tablet
I20240503 02:12:20.413841 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 92) took 1188 ms. Trace:
I20240503 02:12:20.414023 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 110c67212ec345028024da32968869ba (table default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]) successfully deleted
I20240503 02:12:20.413954 5114 rpcz_store.cc:276] 0503 02:12:19.225231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.225369 (+ 138us) service_pool.cc:225] Handling call
0503 02:12:20.413685 (+1188316us) ts_tablet_manager.cc:1120] Deleting tablet 2a51563396614c54b980f0679e84a2b1
0503 02:12:20.413696 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.413805 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172968}
I20240503 02:12:20.414472 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 89) took 1322 ms. Trace:
I20240503 02:12:20.414589 5114 rpcz_store.cc:276] 0503 02:12:19.092061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.215974 (+123913us) service_pool.cc:225] Handling call
0503 02:12:20.414343 (+1198369us) ts_tablet_manager.cc:1120] Deleting tablet ab3c55f80a644cefa87b5769f15bb7f0
0503 02:12:20.414352 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.414440 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143276}
I20240503 02:12:20.414647 5105 raft_consensus.cc:348] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.415004 5105 raft_consensus.cc:374] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.414994 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7b69392fa5f441049437639e143a7491 (table default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]) successfully deleted
I20240503 02:12:20.415033 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 97) took 1188 ms. Trace:
I20240503 02:12:20.415143 5105 raft_consensus.cc:729] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.415139 5114 rpcz_store.cc:276] 0503 02:12:19.226796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.228742 (+ 1946us) service_pool.cc:225] Handling call
0503 02:12:20.414907 (+1186165us) ts_tablet_manager.cc:1120] Deleting tablet ced251fecca84866ab3ff15bd225e2b7
0503 02:12:20.414915 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.414998 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112515}
I20240503 02:12:20.415513 5114 tablet_replica.cc:330] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.415524 5105 consensus_queue.cc:260] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.415728 5114 raft_consensus.cc:2229] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:20.415755 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 003c9c3e0a1d400c94bfaefada2e9faa already in progress: creating tablet
I20240503 02:12:20.415907 5114 raft_consensus.cc:2258] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:20.415947 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 003c9c3e0a1d400c94bfaefada2e9faa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 003c9c3e0a1d400c94bfaefada2e9faa already in progress: creating tablet
I20240503 02:12:20.416124 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet c46ea1084304449e895be230d86471a9 (table default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]) successfully deleted
I20240503 02:12:20.416309 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 461a6b30ab59485ea1a2ef8ae9815e04 (table default.Table_75654fad5307429586e50445bc45687f [id=f1d32bef89ec4f2d83bc96623aef7ef6]) successfully deleted
I20240503 02:12:20.416494 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b487131ac3a842009b3fe54578ea091d (table default.Table_f1fb4997ae544428a3067a3b9ae447dc [id=a24019c2303b40a2adab179b7ea5ca1b]) successfully deleted
W20240503 02:12:20.416677 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2a51563396614c54b980f0679e84a2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a51563396614c54b980f0679e84a2b1 already in progress: creating tablet
I20240503 02:12:20.416776 5114 ts_tablet_manager.cc:1875] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.418303 5105 ts_tablet_manager.cc:1398] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:20.419123 4648 tablet_service.cc:1467] Processing CreateTablet for tablet bbab4368e73b4944b9853c456a4db7bd (DEFAULT_TABLE table=default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.419648 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbab4368e73b4944b9853c456a4db7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.421069 5105 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.423238 5105 tablet_bootstrap.cc:654] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:20.416877 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ab3c55f80a644cefa87b5769f15bb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab3c55f80a644cefa87b5769f15bb7f0 already in progress: creating tablet
W20240503 02:12:20.424973 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ced251fecca84866ab3ff15bd225e2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced251fecca84866ab3ff15bd225e2b7 already in progress: creating tablet
I20240503 02:12:20.425025 5114 ts_tablet_manager.cc:1888] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.425163 5114 log.cc:1196] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/fc887bc4db1c4e268c4c9f23096d704c
I20240503 02:12:20.425537 5114 ts_tablet_manager.cc:1909] T fc887bc4db1c4e268c4c9f23096d704c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.425606 5200 raft_consensus.cc:482] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.425774 5200 raft_consensus.cc:504] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.426416 5200 leader_election.cc:288] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.426596 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.426824 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 617ad14927c1441b800a204aefc2e619 (DEFAULT_TABLE table=default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.426824 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 48135062d92949dc8386c43cf103ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.427073 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 (DEFAULT_TABLE table=default.Table_f663b1d9cbdd47cb82ea64536b0e5013 [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.427508 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 19fb0929bd234788b5313d96d2e4845c (DEFAULT_TABLE table=default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.427556 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 617ad14927c1441b800a204aefc2e619 (DEFAULT_TABLE table=default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.427893 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.427980 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ad14927c1441b800a204aefc2e619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.428187 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 19fb0929bd234788b5313d96d2e4845c (DEFAULT_TABLE table=default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.428336 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 48135062d92949dc8386c43cf103ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.428584 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19fb0929bd234788b5313d96d2e4845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.428536 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.428624 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a3ca705e8482a8cd6a6fb17d93470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.428910 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9327a9e5472b4eda854320c3a95f8e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.429227 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19fb0929bd234788b5313d96d2e4845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.430421 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 96) took 1203 ms. Trace:
I20240503 02:12:20.430509 5114 rpcz_store.cc:276] 0503 02:12:19.226695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.228704 (+ 2009us) service_pool.cc:225] Handling call
0503 02:12:20.415452 (+1186748us) ts_tablet_manager.cc:1120] Deleting tablet fc887bc4db1c4e268c4c9f23096d704c
0503 02:12:20.415460 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.421865 (+ 6405us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.424891 (+ 3026us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.430367 (+ 5476us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1074,"tablet-delete.queue_time_us":1113006}
I20240503 02:12:20.431005 5114 tablet_replica.cc:330] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.431198 5114 raft_consensus.cc:2229] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.431377 5114 raft_consensus.cc:2258] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.431980 5114 ts_tablet_manager.cc:1875] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.432459 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ad14927c1441b800a204aefc2e619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.433283 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 (DEFAULT_TABLE table=default.Table_f663b1d9cbdd47cb82ea64536b0e5013 [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.433539 4652 tablet_service.cc:1467] Processing CreateTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb (DEFAULT_TABLE table=default.Table_b6ddecc9e022479595b1ce5c50323184 [id=e08a62f9895e4721b5524313acf78175]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.433687 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a3ca705e8482a8cd6a6fb17d93470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.433921 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b080fc5ffc564420a0b7f2b7460ebbbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.434258 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 19fb0929bd234788b5313d96d2e4845c (DEFAULT_TABLE table=default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.434592 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19fb0929bd234788b5313d96d2e4845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.434799 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effb2ce36a31477382865461d89917be" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:20.435037 4529 raft_consensus.cc:2454] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:20.437093 5106 tablet_bootstrap.cc:654] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.437283 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (DEFAULT_TABLE table=default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.437736 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.435658 4608 leader_election.cc:341] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:20.441334 5103 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.441510 5103 ts_tablet_manager.cc:1367] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.005s
I20240503 02:12:20.442438 5103 raft_consensus.cc:348] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.442708 5103 raft_consensus.cc:374] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.427590 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.442790 5103 raft_consensus.cc:729] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.443046 5103 consensus_queue.cc:260] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.443586 5103 ts_tablet_manager.cc:1398] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:20.443809 5243 raft_consensus.cc:482] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.443910 5103 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.443994 5243 raft_consensus.cc:504] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.444202 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 (DEFAULT_TABLE table=default.Table_f663b1d9cbdd47cb82ea64536b0e5013 [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.444635 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a3ca705e8482a8cd6a6fb17d93470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.444787 5243 leader_election.cc:288] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:20.445353 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4064ff59de474e0db3a578eae10c3fa6 consensus state has no leader
I20240503 02:12:20.449118 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546afc74e6c54f8aa5f7c684484d610b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:20.449666 4462 leader_election.cc:341] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:20.439451 4608 leader_election.cc:341] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c46ea1084304449e895be230d86471a9
I20240503 02:12:20.451002 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 48135062d92949dc8386c43cf103ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:20.451225 4608 leader_election.cc:341] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:20.451287 5159 tablet_replica.cc:330] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.451339 4608 leader_election.cc:302] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:20.451505 5159 raft_consensus.cc:2229] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:20.451735 4608 leader_election.cc:341] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: c46ea1084304449e895be230d86471a9
I20240503 02:12:20.451833 4608 leader_election.cc:302] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:20.451889 5159 raft_consensus.cc:2258] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.441279 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 09005fb01c384122a7310c1eea12d3f0 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.452236 5103 tablet_bootstrap.cc:654] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.452550 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09005fb01c384122a7310c1eea12d3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.441289 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (DEFAULT_TABLE table=default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.453356 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.453996 5159 ts_tablet_manager.cc:1875] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.441426 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0e75639a4f4f168f9d833b090ae76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.441533 4653 tablet_service.cc:1514] Processing DeleteTablet for tablet a295609170a8482f8373dbf817b1eea1 with delete_type TABLET_DATA_DELETED (replaced by tablet 22776c81d1a24d368abb68274cf32a0b) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.441536 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 617ad14927c1441b800a204aefc2e619 (DEFAULT_TABLE table=default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.457763 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ad14927c1441b800a204aefc2e619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.458293 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 8750b02c5936412db525d2d3e77befeb (DEFAULT_TABLE table=default.Table_05e6a7077cf54c3ebcfb08151b563cdc [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.458647 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 3 got 0
I20240503 02:12:20.458755 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8750b02c5936412db525d2d3e77befeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.458931 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e40cb35a8d544e85ae13377724ed4727 (table default.Table_8a018bc2b9e84350b29ef6447a8bb965 [id=ecdb526541774f2cbecc22c8efc4c3dd]). Expected version 3 got 0
I20240503 02:12:20.441612 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.441722 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.442090 5115 tablet_replica.cc:330] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:20.461591 5115 raft_consensus.cc:2229] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.461844 5115 raft_consensus.cc:2258] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.433279 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc (DEFAULT_TABLE table=default.Table_bba46dea649d4895bacf0532ffeb8a7f [id=5febc608abd84e7ba26288f2e155d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.462787 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17afd88bf6c040ad9e8d571eccd348dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.449940 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546afc74e6c54f8aa5f7c684484d610b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:20.466251 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 396b48865e954d05b9ba8084a2e7f29b (DEFAULT_TABLE table=default.Table_06ee7797a50e4886b1d001c0538b9d50 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.466643 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396b48865e954d05b9ba8084a2e7f29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.467303 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.467705 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.467864 4792 tablet_service.cc:1467] Processing CreateTablet for tablet d022b5df4a914e439fff918f70c34157 (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.468271 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022b5df4a914e439fff918f70c34157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.470036 5159 ts_tablet_manager.cc:1888] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:20.470192 5159 log.cc:1196] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/48135062d92949dc8386c43cf103ced4
I20240503 02:12:20.470654 5159 ts_tablet_manager.cc:1909] T 48135062d92949dc8386c43cf103ced4 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.460889 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f07e8b862b79406bab7c9aa1e63fd2f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.464581 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bc9150faaae244e8bc1953025cc8c19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.473235 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b23be1160c454c139128aff5a960d183 (DEFAULT_TABLE table=default.Table_828411f9b595429a8ca36e4b054c14f1 [id=ae1d450680154e00bb20d21e106dbd53]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:20.473567 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc9150faaae244e8bc1953025cc8c19f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.473619 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23be1160c454c139128aff5a960d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.477026 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.477419 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:20.473572 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.477598 5159 tablet_replica.cc:330] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.477774 5159 raft_consensus.cc:2229] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.477825 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c126b8ef15c5432abf037e28a5eeba58 (table default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]) successfully deleted
I20240503 02:12:20.477963 5159 raft_consensus.cc:2258] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:20.478137 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7d1210a1217247b9801226055e2f2195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1210a1217247b9801226055e2f2195 already in progress: creating tablet
W20240503 02:12:20.478386 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b7daa5af6c8044178b9f84aaba4cca33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7daa5af6c8044178b9f84aaba4cca33 already in progress: creating tablet
W20240503 02:12:20.478600 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9f57f3833a9240c88e0de1b486fae265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f57f3833a9240c88e0de1b486fae265 already in progress: creating tablet
W20240503 02:12:20.479081 4462 leader_election.cc:341] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:20.479142 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet bc0e75639a4f4f168f9d833b090ae76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc0e75639a4f4f168f9d833b090ae76a already in progress: creating tablet
W20240503 02:12:20.479327 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 387a596cfe654aa99e83d070b7884e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 387a596cfe654aa99e83d070b7884e6b already in progress: creating tablet
I20240503 02:12:20.479418 4462 leader_election.cc:302] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:20.479622 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a295609170a8482f8373dbf817b1eea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a295609170a8482f8373dbf817b1eea1 already in progress: creating tablet
I20240503 02:12:20.480074 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7b69392fa5f441049437639e143a7491 (table default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]) successfully deleted
I20240503 02:12:20.480211 5243 raft_consensus.cc:2726] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:20.480295 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 894b753b71a04193905a190da320a260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 894b753b71a04193905a190da320a260 already in progress: creating tablet
W20240503 02:12:20.480685 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b28bb9ac773c4bc685cb8193e574460b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b28bb9ac773c4bc685cb8193e574460b already in progress: creating tablet
I20240503 02:12:20.481055 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_8278654b5129427a9721eed448fc1758 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
W20240503 02:12:20.481330 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 37a54873b48847c39a2b961ab5c3806c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a54873b48847c39a2b961ab5c3806c already in progress: creating tablet
I20240503 02:12:20.481637 5200 raft_consensus.cc:2726] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:20.481865 5200 raft_consensus.cc:2726] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:20.482290 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 8750b02c5936412db525d2d3e77befeb (DEFAULT_TABLE table=default.Table_05e6a7077cf54c3ebcfb08151b563cdc [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.482537 5115 ts_tablet_manager.cc:1875] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.482738 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8750b02c5936412db525d2d3e77befeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.483636 4788 tablet_service.cc:1467] Processing CreateTablet for tablet e94031465ce7407dab0f90ad8809ac21 (DEFAULT_TABLE table=default.Table_631045fb785640fca1c307babc5c2276 [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.484015 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94031465ce7407dab0f90ad8809ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.484687 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 (DEFAULT_TABLE table=default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.485116 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a14c4f785f4514ae74b8a0b3073503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.489240 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41dce88919456aa846dbdcf5e2759f (DEFAULT_TABLE table=default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.489655 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41dce88919456aa846dbdcf5e2759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.490088 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2b80e65a3283458d8556b0832d236304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.490686 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b80e65a3283458d8556b0832d236304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.493278 4784 tablet_service.cc:1467] Processing CreateTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 (DEFAULT_TABLE table=default.Table_a258c9bad19a4ce393deddba14dfcdbc [id=363231755ce84709b926a7416f3bc40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.493669 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d271d2c112ba44b68d2933a6ebfe7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.494019 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053537029a44d5a519b612d09d19a1 (DEFAULT_TABLE table=default.Table_96c3eaec82a44cecbc28140867a68a73 [id=d9011705ac294bd5832490cfdb4582b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.494388 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053537029a44d5a519b612d09d19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.499244 5106 tablet_bootstrap.cc:492] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:20.499411 5106 ts_tablet_manager.cc:1367] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.111s user 0.000s sys 0.004s
W20240503 02:12:20.499934 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d919b9262ac48cbb8b3f893918b5590 has no consensus state
I20240503 02:12:20.500375 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_00ad2435ac2849cd8807b5bfd284c6e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.501298 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_96c3eaec82a44cecbc28140867a68a73" } new_table_name: "default.Table_34d04f474eb84a42948c4bde48e80439" modify_external_catalogs: true
W20240503 02:12:20.502489 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00ad2435ac2849cd8807b5bfd284c6e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.504195 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (DEFAULT_TABLE table=default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.504650 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.508386 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet b487131ac3a842009b3fe54578ea091d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.508893 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd (DEFAULT_TABLE table=default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.509393 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbbfa2d1a49400c809d5b476036cdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.509544 5243 raft_consensus.cc:482] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.509721 5243 raft_consensus.cc:504] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.510186 4271 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b3223b35d26d425582e61f70f17fbdc9
I20240503 02:12:20.511631 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e625e02b764ff2b7279b35dd10b08c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:20.511855 4810 raft_consensus.cc:2454] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:20.511976 5243 leader_election.cc:288] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.512486 4462 leader_election.cc:302] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:20.513305 4508 tablet_service.cc:1467] Processing CreateTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 (DEFAULT_TABLE table=default.Table_a258c9bad19a4ce393deddba14dfcdbc [id=363231755ce84709b926a7416f3bc40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.513741 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d271d2c112ba44b68d2933a6ebfe7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.481844 5159 ts_tablet_manager.cc:1875] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.515527 5243 raft_consensus.cc:2781] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:20.515576 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 396b48865e954d05b9ba8084a2e7f29b (DEFAULT_TABLE table=default.Table_06ee7797a50e4886b1d001c0538b9d50 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.516698 4501 tablet_service.cc:1467] Processing CreateTablet for tablet b23be1160c454c139128aff5a960d183 (DEFAULT_TABLE table=default.Table_828411f9b595429a8ca36e4b054c14f1 [id=ae1d450680154e00bb20d21e106dbd53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.516485 4504 tablet_service.cc:1467] Processing CreateTablet for tablet d022b5df4a914e439fff918f70c34157 (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.518846 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.520332 5243 raft_consensus.cc:482] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:20.520344 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5456c87cd6d8459f9d35100bec3f8ffc" } new_table_name: "default.Table_89d9f0f67e8841b69f9567d09f7729b2" modify_external_catalogs: true
I20240503 02:12:20.520370 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd (DEFAULT_TABLE table=default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.521085 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396b48865e954d05b9ba8084a2e7f29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.521284 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.521548 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23be1160c454c139128aff5a960d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.521733 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbbfa2d1a49400c809d5b476036cdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.523397 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_487c2ed324fb43f0a376681dd950cbad" } new_table_name: "default.Table_23083e3bb77444ec82de47917ee95f01" modify_external_catalogs: true
I20240503 02:12:20.525532 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet bbab4368e73b4944b9853c456a4db7bd
I20240503 02:12:20.526294 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_150a5f1631d94beaa4e6b81e2e9ade3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.526901 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.527653 5159 ts_tablet_manager.cc:1888] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:20.528883 4259 consensus_peers.cc:485] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d -> Peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Couldn't send request to peer 00c621d795c44d0daa534261920d874a. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:20.531424 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:20.529275 4782 tablet_service.cc:1467] Processing CreateTablet for tablet d19ed19da5b74583bfc3b21d88e937c9 (DEFAULT_TABLE table=default.Table_4e4a5e92b87343dc9067868e6ba38292 [id=fd26a1e3d27a40e7a3b0172d751d6c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.533360 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19ed19da5b74583bfc3b21d88e937c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.529527 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (DEFAULT_TABLE table=default.Table_a95361cb35f04f1fb5b9e5ff04da58a1 [id=ad0169357ed4491c92e35a64d6a202bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.529767 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56701ed6424c60bf1a985eea1936fe (DEFAULT_TABLE table=default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.530066 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (DEFAULT_TABLE table=default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.530846 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.530762 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 001fe4bb80c24d8e97d971a7130b2296 (DEFAULT_TABLE table=default.Table_1f7ba66d347245859fec5ef1be88f98f [id=fbc72fd4039b499bbce796cad96e1be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.536996 5105 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.530542 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea1eb32394b04b72258d24196202b (DEFAULT_TABLE table=default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.537691 5105 ts_tablet_manager.cc:1367] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.002s
I20240503 02:12:20.537374 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d58f97b58481ea1dfa75397a32ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.538091 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.538357 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56701ed6424c60bf1a985eea1936fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.538580 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41dce88919456aa846dbdcf5e2759f (DEFAULT_TABLE table=default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.538975 5243 raft_consensus.cc:3037] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.538539 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053537029a44d5a519b612d09d19a1 (DEFAULT_TABLE table=default.Table_96c3eaec82a44cecbc28140867a68a73 [id=d9011705ac294bd5832490cfdb4582b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.531297 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 53017352b8f34955b116bbfd32f04854 (DEFAULT_TABLE table=default.Table_0799f0ea89c14b95b8593cd2e1423759 [id=47856b6b36f74b5589f9acc60162c1e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.539417 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.539369 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 (DEFAULT_TABLE table=default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.532860 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.529290 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 22776c81d1a24d368abb68274cf32a0b (DEFAULT_TABLE table=default.Table_dde3e4c146264ea39a885cb77419db45 [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.539975 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.540684 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001fe4bb80c24d8e97d971a7130b2296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.540899 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea1eb32394b04b72258d24196202b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.540899 4491 tablet_service.cc:1467] Processing CreateTablet for tablet e94031465ce7407dab0f90ad8809ac21 (DEFAULT_TABLE table=default.Table_631045fb785640fca1c307babc5c2276 [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.541231 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022b5df4a914e439fff918f70c34157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.547719 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea1eb32394b04b72258d24196202b (DEFAULT_TABLE table=default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.548364 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41dce88919456aa846dbdcf5e2759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.548640 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22776c81d1a24d368abb68274cf32a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.549285 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a14c4f785f4514ae74b8a0b3073503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.550146 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53017352b8f34955b116bbfd32f04854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.549976 5159 log.cc:1196] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/110c67212ec345028024da32968869ba
I20240503 02:12:20.556800 5159 ts_tablet_manager.cc:1909] T 110c67212ec345028024da32968869ba P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.557258 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94031465ce7407dab0f90ad8809ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.559525 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e625e02b764ff2b7279b35dd10b08c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:20.560707 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56701ed6424c60bf1a985eea1936fe (DEFAULT_TABLE table=default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.561118 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 53017352b8f34955b116bbfd32f04854 (DEFAULT_TABLE table=default.Table_0799f0ea89c14b95b8593cd2e1423759 [id=47856b6b36f74b5589f9acc60162c1e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.564705 5105 raft_consensus.cc:348] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.564961 4494 tablet_service.cc:1467] Processing CreateTablet for tablet d19ed19da5b74583bfc3b21d88e937c9 (DEFAULT_TABLE table=default.Table_4e4a5e92b87343dc9067868e6ba38292 [id=fd26a1e3d27a40e7a3b0172d751d6c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.566699 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 110c67212ec345028024da32968869ba (table default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]) successfully deleted
I20240503 02:12:20.566933 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (DEFAULT_TABLE table=default.Table_a95361cb35f04f1fb5b9e5ff04da58a1 [id=ad0169357ed4491c92e35a64d6a202bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:20.568200 4462 leader_election.cc:341] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.565665 5243 raft_consensus.cc:504] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.569293 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (DEFAULT_TABLE table=default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.570585 5159 tablet_replica.cc:330] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.571103 5159 raft_consensus.cc:2229] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.571291 5159 raft_consensus.cc:2258] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.571462 5105 raft_consensus.cc:374] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.571568 5105 raft_consensus.cc:729] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.571792 5105 consensus_queue.cc:260] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.572399 5105 ts_tablet_manager.cc:1398] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 02:12:20.572685 5105 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.573413 5159 ts_tablet_manager.cc:1875] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.569171 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 22776c81d1a24d368abb68274cf32a0b (DEFAULT_TABLE table=default.Table_dde3e4c146264ea39a885cb77419db45 [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.577773 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea1eb32394b04b72258d24196202b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.575251 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 001fe4bb80c24d8e97d971a7130b2296 (DEFAULT_TABLE table=default.Table_1f7ba66d347245859fec5ef1be88f98f [id=fbc72fd4039b499bbce796cad96e1be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.579409 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e625e02b764ff2b7279b35dd10b08c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:20.582496 5243 leader_election.cc:288] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.583772 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet a295609170a8482f8373dbf817b1eea1 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.584342 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:20.583249 4462 leader_election.cc:341] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:20.586513 5105 tablet_bootstrap.cc:654] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.587622 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56701ed6424c60bf1a985eea1936fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.589424 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.590528 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53017352b8f34955b116bbfd32f04854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.591110 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053537029a44d5a519b612d09d19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.591842 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d58f97b58481ea1dfa75397a32ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.595180 5159 ts_tablet_manager.cc:1888] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.595158 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_150d9a5a76ef45ba89eae071bd04eab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.595547 5159 log.cc:1196] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/21e625e02b764ff2b7279b35dd10b08c
W20240503 02:12:20.595810 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150d9a5a76ef45ba89eae071bd04eab6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.595950 5159 ts_tablet_manager.cc:1909] T 21e625e02b764ff2b7279b35dd10b08c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.597523 5159 tablet_replica.cc:330] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.597985 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 21e625e02b764ff2b7279b35dd10b08c (table default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]) successfully deleted
I20240503 02:12:20.598696 5159 raft_consensus.cc:2229] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.598932 5103 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.596485 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet fc887bc4db1c4e268c4c9f23096d704c with delete_type TABLET_DATA_DELETED (replaced by tablet 894b753b71a04193905a190da320a260) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.599165 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.599550 5159 raft_consensus.cc:2258] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.600734 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001fe4bb80c24d8e97d971a7130b2296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.605602 5105 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.605816 5105 ts_tablet_manager.cc:1367] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20240503 02:12:20.606585 5105 raft_consensus.cc:348] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.606850 5105 raft_consensus.cc:374] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.606947 5105 raft_consensus.cc:729] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.607229 5105 consensus_queue.cc:260] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.607782 5105 ts_tablet_manager.cc:1398] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:20.608587 5105 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.610677 5105 tablet_bootstrap.cc:654] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.610733 5159 ts_tablet_manager.cc:1875] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.614583 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8fb97a4f46484d98a7e0d2d19788c5fb" } new_table_name: "default.Table_11cc9c0da9b14e2dab0e03b3373ad949" modify_external_catalogs: true
I20240503 02:12:20.615453 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e625e02b764ff2b7279b35dd10b08c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:20.618499 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:20.618752 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:20.649123 5106 raft_consensus.cc:348] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.649978 5106 raft_consensus.cc:374] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.650384 5106 raft_consensus.cc:729] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.651594 5105 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.651191 5106 consensus_queue.cc:260] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.653085 5105 ts_tablet_manager.cc:1367] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.015s
I20240503 02:12:20.653364 5106 ts_tablet_manager.cc:1398] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.154s user 0.001s sys 0.003s
I20240503 02:12:20.654425 5105 raft_consensus.cc:348] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.654791 5106 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:20.657033 5105 raft_consensus.cc:374] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.657230 5105 raft_consensus.cc:729] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.657297 5115 ts_tablet_manager.cc:1888] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:20.657691 5115 log.cc:1196] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/48135062d92949dc8386c43cf103ced4
I20240503 02:12:20.658164 5105 consensus_queue.cc:260] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.658771 5105 ts_tablet_manager.cc:1398] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:20.659102 5105 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.666492 5115 ts_tablet_manager.cc:1909] T 48135062d92949dc8386c43cf103ced4 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:20.667279 4462 leader_election.cc:341] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.668140 5105 tablet_bootstrap.cc:654] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.656886 5159 ts_tablet_manager.cc:1888] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.669135 5159 log.cc:1196] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b487131ac3a842009b3fe54578ea091d
I20240503 02:12:20.669536 5159 ts_tablet_manager.cc:1909] T b487131ac3a842009b3fe54578ea091d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.670892 5159 tablet_replica.cc:330] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.671128 5159 raft_consensus.cc:2229] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.671317 5159 raft_consensus.cc:2258] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.674645 5159 ts_tablet_manager.cc:1875] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.681349 5159 ts_tablet_manager.cc:1888] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.681547 5159 log.cc:1196] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fc887bc4db1c4e268c4c9f23096d704c
I20240503 02:12:20.681900 5159 ts_tablet_manager.cc:1909] T fc887bc4db1c4e268c4c9f23096d704c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.687336 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b487131ac3a842009b3fe54578ea091d (table default.Table_f1fb4997ae544428a3067a3b9ae447dc [id=a24019c2303b40a2adab179b7ea5ca1b]) successfully deleted
W20240503 02:12:20.689569 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet bbab4368e73b4944b9853c456a4db7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbab4368e73b4944b9853c456a4db7bd already in progress: creating tablet
W20240503 02:12:20.691147 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd402c0eb9e4cf49dc722f2dd214e09 already in progress: creating tablet
I20240503 02:12:20.692765 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fc887bc4db1c4e268c4c9f23096d704c (table default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]) successfully deleted
I20240503 02:12:20.671651 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19ed19da5b74583bfc3b21d88e937c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.675704 4462 leader_election.cc:302] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:20.670348 5103 ts_tablet_manager.cc:1367] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.156s user 0.015s sys 0.010s
I20240503 02:12:20.729818 5103 raft_consensus.cc:348] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.730068 5103 raft_consensus.cc:374] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.730163 5103 raft_consensus.cc:729] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.731724 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22776c81d1a24d368abb68274cf32a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.732029 5106 tablet_bootstrap.cc:654] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:20.741330 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 289551us
W20240503 02:12:20.741734 4608 leader_election.cc:341] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 634beeb9d8394b1da8c4ad79ba7e26ac
I20240503 02:12:20.742019 4608 leader_election.cc:302] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:20.745268 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 129188us
I20240503 02:12:20.745574 5243 raft_consensus.cc:2726] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:20.749105 5256 raft_consensus.cc:2781] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:20.749253 5256 raft_consensus.cc:482] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:20.749363 5256 raft_consensus.cc:3037] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.752275 5256 raft_consensus.cc:504] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.753608 5256 leader_election.cc:288] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:20.754098 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 387a596cfe654aa99e83d070b7884e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 387a596cfe654aa99e83d070b7884e6b already in progress: creating tablet
I20240503 02:12:20.756578 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_8278654b5129427a9721eed448fc1758 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
W20240503 02:12:20.757020 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet bc0e75639a4f4f168f9d833b090ae76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc0e75639a4f4f168f9d833b090ae76a already in progress: creating tablet
W20240503 02:12:20.757406 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a295609170a8482f8373dbf817b1eea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a295609170a8482f8373dbf817b1eea1 already in progress: creating tablet
W20240503 02:12:20.757771 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 894b753b71a04193905a190da320a260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 894b753b71a04193905a190da320a260 already in progress: creating tablet
W20240503 02:12:20.758098 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b28bb9ac773c4bc685cb8193e574460b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b28bb9ac773c4bc685cb8193e574460b already in progress: creating tablet
I20240503 02:12:20.759972 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effb2ce36a31477382865461d89917be" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:20.760213 4529 raft_consensus.cc:3037] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.763484 4529 raft_consensus.cc:2454] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:20.730394 5103 consensus_queue.cc:260] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.766412 5103 ts_tablet_manager.cc:1398] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240503 02:12:20.767495 5114 ts_tablet_manager.cc:1888] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.767644 5114 log.cc:1196] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/851756dc1a5942919c85967f83eca29a
I20240503 02:12:20.768081 5114 ts_tablet_manager.cc:1909] T 851756dc1a5942919c85967f83eca29a P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:20.771584 5103 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.779183 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 396b48865e954d05b9ba8084a2e7f29b (DEFAULT_TABLE table=default.Table_06ee7797a50e4886b1d001c0538b9d50 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.779728 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396b48865e954d05b9ba8084a2e7f29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.781018 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.781289 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b23be1160c454c139128aff5a960d183 (DEFAULT_TABLE table=default.Table_828411f9b595429a8ca36e4b054c14f1 [id=ae1d450680154e00bb20d21e106dbd53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.781347 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 8750b02c5936412db525d2d3e77befeb (DEFAULT_TABLE table=default.Table_05e6a7077cf54c3ebcfb08151b563cdc [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.781746 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23be1160c454c139128aff5a960d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.781955 4636 tablet_service.cc:1467] Processing CreateTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 (DEFAULT_TABLE table=default.Table_a258c9bad19a4ce393deddba14dfcdbc [id=363231755ce84709b926a7416f3bc40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.782321 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d271d2c112ba44b68d2933a6ebfe7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.783002 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.783449 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet f22a4b8015354db493652d8f336a1265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.784096 4648 tablet_service.cc:1467] Processing CreateTablet for tablet e94031465ce7407dab0f90ad8809ac21 (DEFAULT_TABLE table=default.Table_631045fb785640fca1c307babc5c2276 [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.784502 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94031465ce7407dab0f90ad8809ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.785264 4652 tablet_service.cc:1467] Processing CreateTablet for tablet d022b5df4a914e439fff918f70c34157 (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.785650 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022b5df4a914e439fff918f70c34157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.792425 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8750b02c5936412db525d2d3e77befeb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.793378 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f053537029a44d5a519b612d09d19a1 consensus state has no leader
I20240503 02:12:20.794848 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_27d1ed70e95c4b648e63e37a036f1b42" } modify_external_catalogs: true
I20240503 02:12:20.795042 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet d989197a5d6e42379df6d8add449a913 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.797174 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053537029a44d5a519b612d09d19a1 (DEFAULT_TABLE table=default.Table_96c3eaec82a44cecbc28140867a68a73 [id=d9011705ac294bd5832490cfdb4582b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.797734 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053537029a44d5a519b612d09d19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.799252 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 387a596cfe654aa99e83d070b7884e6b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.799930 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_464b5c2e54804303a2039f77c6a57dd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.800521 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_464b5c2e54804303a2039f77c6a57dd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.802502 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 (DEFAULT_TABLE table=default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.802677 5105 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.802870 5105 ts_tablet_manager.cc:1367] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.144s user 0.020s sys 0.036s
I20240503 02:12:20.802932 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a14c4f785f4514ae74b8a0b3073503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.804296 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41dce88919456aa846dbdcf5e2759f (DEFAULT_TABLE table=default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.804787 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41dce88919456aa846dbdcf5e2759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.805275 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd (DEFAULT_TABLE table=default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.805802 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbbfa2d1a49400c809d5b476036cdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.804438 5106 tablet_bootstrap.cc:492] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:20.803702 5105 raft_consensus.cc:348] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.806820 5105 raft_consensus.cc:374] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.806914 5105 raft_consensus.cc:729] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.807169 5105 consensus_queue.cc:260] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.807562 5103 tablet_bootstrap.cc:654] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.807744 5105 ts_tablet_manager.cc:1398] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:20.808097 5105 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:20.808113 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea1eb32394b04b72258d24196202b (DEFAULT_TABLE table=default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.809182 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 53017352b8f34955b116bbfd32f04854 (DEFAULT_TABLE table=default.Table_0799f0ea89c14b95b8593cd2e1423759 [id=47856b6b36f74b5589f9acc60162c1e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.810335 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56701ed6424c60bf1a985eea1936fe (DEFAULT_TABLE table=default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.810895 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (DEFAULT_TABLE table=default.Table_a95361cb35f04f1fb5b9e5ff04da58a1 [id=ad0169357ed4491c92e35a64d6a202bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.811296 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 22776c81d1a24d368abb68274cf32a0b (DEFAULT_TABLE table=default.Table_dde3e4c146264ea39a885cb77419db45 [id=07ea56ce8d3749dfbcbc1a79ea84693c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.813935 4644 tablet_service.cc:1467] Processing CreateTablet for tablet d19ed19da5b74583bfc3b21d88e937c9 (DEFAULT_TABLE table=default.Table_4e4a5e92b87343dc9067868e6ba38292 [id=fd26a1e3d27a40e7a3b0172d751d6c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.815017 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 001fe4bb80c24d8e97d971a7130b2296 (DEFAULT_TABLE table=default.Table_1f7ba66d347245859fec5ef1be88f98f [id=fbc72fd4039b499bbce796cad96e1be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.816071 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.795042 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_27d1ed70e95c4b648e63e37a036f1b42" } modify_external_catalogs: true
W20240503 02:12:20.817843 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64cdb861bc9f448698e243718136c303 consensus state has no leader
I20240503 02:12:20.818011 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22776c81d1a24d368abb68274cf32a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.818715 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--webserver_interface=127.4.17.126
--webserver_port=35675
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:12:20.819666 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 53dad4e45a184dc1ae4a82021a48e9be from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.820147 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea1eb32394b04b72258d24196202b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.822057 5105 tablet_bootstrap.cc:654] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.822705 5106 ts_tablet_manager.cc:1367] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.003s
I20240503 02:12:20.823632 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (DEFAULT_TABLE table=default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.823977 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001fe4bb80c24d8e97d971a7130b2296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.824851 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.825074 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.825316 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19ed19da5b74583bfc3b21d88e937c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.831378 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 634beeb9d8394b1da8c4ad79ba7e26ac with delete_type TABLET_DATA_DELETED (replaced by tablet ced251fecca84866ab3ff15bd225e2b7) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.831466 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d58f97b58481ea1dfa75397a32ae2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.838726 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46560ac6d94943d2a8fb3b3904f2b5b6 consensus state has no leader
I20240503 02:12:20.840363 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3f5c5ee57b8c4f27987de424a90281cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.841009 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5c5ee57b8c4f27987de424a90281cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.842550 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0e75639a4f4f168f9d833b090ae76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.842875 5159 tablet_replica.cc:330] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.843086 5159 raft_consensus.cc:2229] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.843322 5159 raft_consensus.cc:2258] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.844189 5159 ts_tablet_manager.cc:1875] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.844259 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 90) took 1629 ms. Trace:
I20240503 02:12:20.844455 5114 rpcz_store.cc:276] 0503 02:12:19.214888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.217510 (+ 2622us) service_pool.cc:225] Handling call
0503 02:12:20.430929 (+1213419us) ts_tablet_manager.cc:1120] Deleting tablet 851756dc1a5942919c85967f83eca29a
0503 02:12:20.430938 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.559241 (+128303us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.767274 (+208033us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.844136 (+ 76862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125793}
I20240503 02:12:20.845307 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 108) took 1300 ms. Trace:
I20240503 02:12:20.845536 5114 rpcz_store.cc:276] 0503 02:12:19.545203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.545566 (+ 363us) service_pool.cc:225] Handling call
0503 02:12:20.844985 (+1299419us) ts_tablet_manager.cc:1120] Deleting tablet fe5117d4d59b4f0b87c32e8b27b085d9
0503 02:12:20.845024 (+ 39us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.845243 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299199}
I20240503 02:12:20.846421 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 109) took 1301 ms. Trace:
I20240503 02:12:20.846611 5114 rpcz_store.cc:276] 0503 02:12:19.545322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.545826 (+ 504us) service_pool.cc:225] Handling call
0503 02:12:20.846185 (+1300359us) ts_tablet_manager.cc:1120] Deleting tablet c823155b774e42a5a422994903eae167
0503 02:12:20.846197 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.846387 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285123}
I20240503 02:12:20.847343 5114 tablet_replica.cc:330] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.847772 5114 raft_consensus.cc:2229] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.848109 5114 raft_consensus.cc:2258] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.850675 5159 ts_tablet_manager.cc:1888] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.850838 5159 log.cc:1196] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/bc0e75639a4f4f168f9d833b090ae76a
I20240503 02:12:20.851224 5159 ts_tablet_manager.cc:1909] T bc0e75639a4f4f168f9d833b090ae76a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.855280 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab (DEFAULT_TABLE table=default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.855525 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:20.855881 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b72c20a4b5a4d2ea266f031733d1aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.858829 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab (DEFAULT_TABLE table=default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.859301 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b72c20a4b5a4d2ea266f031733d1aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.861932 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d (DEFAULT_TABLE table=default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.862426 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885129a4efc5475b97b7ac68e1db5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.862874 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d (DEFAULT_TABLE table=default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.863286 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885129a4efc5475b97b7ac68e1db5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.865823 5103 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.866043 5103 ts_tablet_manager.cc:1367] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.002s
I20240503 02:12:20.866950 5103 raft_consensus.cc:348] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.867187 5103 raft_consensus.cc:374] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.867292 5103 raft_consensus.cc:729] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.867560 5103 consensus_queue.cc:260] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.870165 4782 tablet_service.cc:1467] Processing CreateTablet for tablet b161628065704c798020edbe098844c2 (DEFAULT_TABLE table=default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.870311 4503 tablet_service.cc:1467] Processing CreateTablet for tablet b161628065704c798020edbe098844c2 (DEFAULT_TABLE table=default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.870662 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b161628065704c798020edbe098844c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.871811 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7d919b9262ac48cbb8b3f893918b5590 (DEFAULT_TABLE table=default.Table_bbf1abd1706f4e85a126f2c232a55a8b [id=819cdd542d91449db9dd3f7a0c668d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.872644 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet bc0e75639a4f4f168f9d833b090ae76a (table default.Table_5c5ea1ab22594d6b81fbd5b536231007 [id=9947c6f15d2444ca895eea2f96df51ae]) successfully deleted
I20240503 02:12:20.873257 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 851756dc1a5942919c85967f83eca29a (table default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]) successfully deleted
W20240503 02:12:20.873725 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fe5117d4d59b4f0b87c32e8b27b085d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5117d4d59b4f0b87c32e8b27b085d9 already in progress: creating tablet
W20240503 02:12:20.874327 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c823155b774e42a5a422994903eae167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c823155b774e42a5a422994903eae167 already in progress: creating tablet
I20240503 02:12:20.875720 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.876735 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.877286 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 546afc74e6c54f8aa5f7c684484d610b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.877566 5159 tablet_replica.cc:330] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.877787 5159 raft_consensus.cc:2229] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.877969 5159 raft_consensus.cc:2258] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.878924 5159 ts_tablet_manager.cc:1875] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.879556 5103 ts_tablet_manager.cc:1398] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:12:20.880060 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 7d919b9262ac48cbb8b3f893918b5590 (DEFAULT_TABLE table=default.Table_bbf1abd1706f4e85a126f2c232a55a8b [id=819cdd542d91449db9dd3f7a0c668d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.880515 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d919b9262ac48cbb8b3f893918b5590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.886229 5159 ts_tablet_manager.cc:1888] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.886502 5159 log.cc:1196] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/546afc74e6c54f8aa5f7c684484d610b
W20240503 02:12:20.886570 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c9db92b88674d908b1dea62e581caac consensus state has no leader
I20240503 02:12:20.886896 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 09005fb01c384122a7310c1eea12d3f0 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:20.886969 5159 ts_tablet_manager.cc:1909] T 546afc74e6c54f8aa5f7c684484d610b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:20.891106 5114 ts_tablet_manager.cc:1875] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.891734 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b161628065704c798020edbe098844c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.893957 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a210808781ce413f96c49311edbe1415" } modify_external_catalogs: true
I20240503 02:12:20.894217 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a210808781ce413f96c49311edbe1415" } modify_external_catalogs: true
I20240503 02:12:20.894498 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cf42415d1d604ff283f76a5ac98e7347"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.895134 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf42415d1d604ff283f76a5ac98e7347 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.896023 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_31852ddf01b44f4a88e80b4216b590c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.896797 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31852ddf01b44f4a88e80b4216b590c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.897266 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d919b9262ac48cbb8b3f893918b5590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.897305 5243 raft_consensus.cc:482] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.897472 5243 raft_consensus.cc:504] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:20.898252 5243 leader_election.cc:288] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:20.899099 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0799f0ea89c14b95b8593cd2e1423759" } new_table_name: "default.Table_bb18c51dac8c46b2a5ddecca6ddd34be" modify_external_catalogs: true
I20240503 02:12:20.899163 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a829c48a5c24d6d9e6329ac34151d1f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:20.899832 4462 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:20.904000 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_489fd4b2341e48ee904ccad1bdad6d98" } modify_external_catalogs: true
I20240503 02:12:20.904214 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_489fd4b2341e48ee904ccad1bdad6d98" } modify_external_catalogs: true
I20240503 02:12:20.908780 5103 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.915693 5103 tablet_bootstrap.cc:654] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.916639 5256 raft_consensus.cc:482] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.917593 5115 tablet_replica.cc:330] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:20.917586 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5ac4a60d6df343668e711966d75f932c" } modify_external_catalogs: true
I20240503 02:12:20.917780 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5ac4a60d6df343668e711966d75f932c" } modify_external_catalogs: true
I20240503 02:12:20.921864 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56701ed6424c60bf1a985eea1936fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.924152 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet c46ea1084304449e895be230d86471a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.928310 4792 tablet_service.cc:1467] Processing CreateTablet for tablet ea4dac17dca3442d8934ac357779b78a (DEFAULT_TABLE table=default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd [id=89060d0da21a4cf8b3d3c1ab1a8a3749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.928903 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4dac17dca3442d8934ac357779b78a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.929384 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8a829c48a5c24d6d9e6329ac34151d1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a829c48a5c24d6d9e6329ac34151d1f already in progress: creating tablet
I20240503 02:12:20.930243 5256 raft_consensus.cc:504] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.931015 5256 leader_election.cc:288] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:20.932984 5115 raft_consensus.cc:2229] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.933207 5115 raft_consensus.cc:2258] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.937844 5103 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:20.938095 5103 ts_tablet_manager.cc:1367] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.006s
I20240503 02:12:20.938486 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 53dad4e45a184dc1ae4a82021a48e9be
I20240503 02:12:20.938956 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 53dad4e45a184dc1ae4a82021a48e9be deleted and replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:20.939411 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b16c409173a84334895e0c40773793c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:20.940006 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b16c409173a84334895e0c40773793c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.941907 5103 raft_consensus.cc:348] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.942108 5103 raft_consensus.cc:374] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.942988 5103 raft_consensus.cc:729] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.943248 5103 consensus_queue.cc:260] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:20.944015 5103 ts_tablet_manager.cc:1398] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:20.944330 5103 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:20.946614 5103 tablet_bootstrap.cc:654] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.947644 5115 ts_tablet_manager.cc:1875] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.947789 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fee5bceac941cd83471e5cad5dd463 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.950014 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d989197a5d6e42379df6d8add449a913
I20240503 02:12:20.950498 4274 catalog_manager.cc:6554] ReplaceTablet: tablet d989197a5d6e42379df6d8add449a913 deleted and replaced by tablet ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:20.951139 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 617ad14927c1441b800a204aefc2e619
I20240503 02:12:20.951793 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 09005fb01c384122a7310c1eea12d3f0
I20240503 02:12:20.952392 4273 catalog_manager.cc:6554] ReplaceTablet: tablet 09005fb01c384122a7310c1eea12d3f0 deleted and replaced by tablet 41d44fe8f6334572996665bdec60e2e5
I20240503 02:12:20.960011 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 546afc74e6c54f8aa5f7c684484d610b (table default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]) successfully deleted
I20240503 02:12:20.960574 4503 tablet_service.cc:1467] Processing CreateTablet for tablet ea4dac17dca3442d8934ac357779b78a (DEFAULT_TABLE table=default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd [id=89060d0da21a4cf8b3d3c1ab1a8a3749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.962481 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.962850 5159 tablet_replica.cc:330] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:20.962206 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4dac17dca3442d8934ac357779b78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.963058 5159 raft_consensus.cc:2229] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.963269 5159 raft_consensus.cc:2258] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:20.962916 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.964462 5159 ts_tablet_manager.cc:1875] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.964601 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.965484 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.965962 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.966823 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.967238 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.968957 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.970654 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:20.977576 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.978459 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.979215 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.979830 5106 raft_consensus.cc:348] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:20.980190 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53017352b8f34955b116bbfd32f04854 consensus state has no leader
I20240503 02:12:20.981333 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:20.982545 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.986394 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6681572bb5e5469bb236a2eb7975fd0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.986563 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:20.987010 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6681572bb5e5469bb236a2eb7975fd0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.987465 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 19fb0929bd234788b5313d96d2e4845c
I20240503 02:12:20.989691 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:20.991137 5114 ts_tablet_manager.cc:1888] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:20.991302 5114 log.cc:1196] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:20.991807 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:20.991852 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.992242 5114 ts_tablet_manager.cc:1909] T 14fd4fb88436498999b5f339052ac0b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:20.992616 4608 leader_election.cc:341] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: effb2ce36a31477382865461d89917be
I20240503 02:12:20.993140 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:20.993268 4638 tablet_service.cc:1467] Processing CreateTablet for tablet b161628065704c798020edbe098844c2 (DEFAULT_TABLE table=default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.993870 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 7d919b9262ac48cbb8b3f893918b5590 (DEFAULT_TABLE table=default.Table_bbf1abd1706f4e85a126f2c232a55a8b [id=819cdd542d91449db9dd3f7a0c668d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.993149 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 107) took 1448 ms. Trace:
I20240503 02:12:20.995810 5114 rpcz_store.cc:276] 0503 02:12:19.545056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.546764 (+ 1708us) service_pool.cc:225] Handling call
0503 02:12:20.847210 (+1300446us) ts_tablet_manager.cc:1120] Deleting tablet 14fd4fb88436498999b5f339052ac0b6
0503 02:12:20.847219 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:20.969785 (+122566us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.990998 (+ 21213us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.993078 (+ 2080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285040}
I20240503 02:12:20.995961 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:20.996193 5114 tablet_replica.cc:330] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:20.996384 5114 raft_consensus.cc:2229] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:20.996604 5114 raft_consensus.cc:2258] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:20.996649 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:20.997308 5114 ts_tablet_manager.cc:1875] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:20.997310 5105 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:20.997611 5105 ts_tablet_manager.cc:1367] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.190s user 0.050s sys 0.073s
I20240503 02:12:20.999768 5105 raft_consensus.cc:348] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.000207 5105 raft_consensus.cc:374] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.000370 5105 raft_consensus.cc:729] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.001024 5105 consensus_queue.cc:260] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.002115 5105 ts_tablet_manager.cc:1398] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:21.002864 5105 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.007133 5105 tablet_bootstrap.cc:654] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.007790 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dde3e4c146264ea39a885cb77419db45" } new_table_name: "default.Table_98ecda629a0e48c3bcb69d91efbfe2f4" modify_external_catalogs: true
I20240503 02:12:21.008090 5114 ts_tablet_manager.cc:1888] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.008246 5114 log.cc:1196] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/546afc74e6c54f8aa5f7c684484d610b
I20240503 02:12:21.008621 5114 ts_tablet_manager.cc:1909] T 546afc74e6c54f8aa5f7c684484d610b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.009447 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 111) took 1463 ms. Trace:
I20240503 02:12:21.009498 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_827711fb4f7843b88f3481b51b2ee727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.009649 5114 rpcz_store.cc:276] 0503 02:12:19.545513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.546805 (+ 1292us) service_pool.cc:225] Handling call
0503 02:12:20.996102 (+1449297us) ts_tablet_manager.cc:1120] Deleting tablet 546afc74e6c54f8aa5f7c684484d610b
0503 02:12:20.996114 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:21.004746 (+ 8632us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:21.007936 (+ 3190us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:21.009409 (+ 1473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433751}
I20240503 02:12:21.010094 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 110) took 1464 ms. Trace:
I20240503 02:12:21.010078 4646 tablet_service.cc:1467] Processing CreateTablet for tablet ea4dac17dca3442d8934ac357779b78a (DEFAULT_TABLE table=default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd [id=89060d0da21a4cf8b3d3c1ab1a8a3749]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:21.010207 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_827711fb4f7843b88f3481b51b2ee727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.010192 5114 rpcz_store.cc:276] 0503 02:12:19.545422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.545902 (+ 480us) service_pool.cc:225] Handling call
0503 02:12:21.009944 (+1464042us) ts_tablet_manager.cc:1120] Deleting tablet bf9c07c7cbb743d4a8e9610281a00f94
0503 02:12:21.009954 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:21.010059 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441365}
I20240503 02:12:21.011387 5114 tablet_replica.cc:330] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.011595 5114 raft_consensus.cc:2229] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.011755 5114 raft_consensus.cc:2258] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.012295 5114 ts_tablet_manager.cc:1875] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.014781 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.014833 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.020375 5114 ts_tablet_manager.cc:1888] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.020548 5114 log.cc:1196] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/bc0e75639a4f4f168f9d833b090ae76a
I20240503 02:12:21.020900 5114 ts_tablet_manager.cc:1909] T bc0e75639a4f4f168f9d833b090ae76a P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.021508 5105 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.021865 5105 ts_tablet_manager.cc:1367] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20240503 02:12:21.022356 5114 tablet_replica.cc:330] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.022573 5114 raft_consensus.cc:2229] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.022748 5114 raft_consensus.cc:2258] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.023619 5114 ts_tablet_manager.cc:1875] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.023566 5105 raft_consensus.cc:348] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.024135 5105 raft_consensus.cc:374] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.024312 5105 raft_consensus.cc:729] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.024847 5105 consensus_queue.cc:260] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.026119 5105 ts_tablet_manager.cc:1398] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:21.026808 5105 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.027287 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.030042 5105 tablet_bootstrap.cc:654] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.033134 5114 ts_tablet_manager.cc:1888] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.033305 5114 log.cc:1196] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/a295609170a8482f8373dbf817b1eea1
I20240503 02:12:21.033684 5114 ts_tablet_manager.cc:1909] T a295609170a8482f8373dbf817b1eea1 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.035678 5114 tablet_replica.cc:330] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.035919 5114 raft_consensus.cc:2229] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.036337 5201 raft_consensus.cc:482] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.036532 5201 raft_consensus.cc:504] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.037365 5201 leader_election.cc:288] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:21.037942 5114 raft_consensus.cc:2258] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.038244 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:21.038493 4529 raft_consensus.cc:2454] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:21.038938 4746 leader_election.cc:302] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:21.039331 5201 raft_consensus.cc:2781] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:21.039477 5201 raft_consensus.cc:482] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.039597 5201 raft_consensus.cc:3037] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.041003 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 14fd4fb88436498999b5f339052ac0b6 (table default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]) successfully deleted
I20240503 02:12:21.041280 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 546afc74e6c54f8aa5f7c684484d610b (table default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]) successfully deleted
W20240503 02:12:21.041504 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet bf9c07c7cbb743d4a8e9610281a00f94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf9c07c7cbb743d4a8e9610281a00f94 already in progress: creating tablet
I20240503 02:12:21.041517 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:21.041743 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet acdefb33fe6e4a9893dd66f6252395d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acdefb33fe6e4a9893dd66f6252395d7 already in progress: creating tablet
I20240503 02:12:21.041787 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 851756dc1a5942919c85967f83eca29a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:21.041947 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7d5e627d818245e0bdeb16531499f7d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d5e627d818245e0bdeb16531499f7d5 already in progress: creating tablet
W20240503 02:12:21.042162 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 27e8c98d5ea84179a91fdbea8657d6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27e8c98d5ea84179a91fdbea8657d6e6 already in progress: creating tablet
W20240503 02:12:21.042363 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7d1210a1217247b9801226055e2f2195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1210a1217247b9801226055e2f2195 already in progress: creating tablet
W20240503 02:12:21.042585 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9f57f3833a9240c88e0de1b486fae265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f57f3833a9240c88e0de1b486fae265 already in progress: creating tablet
I20240503 02:12:21.042773 5114 ts_tablet_manager.cc:1875] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:21.042801 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b7daa5af6c8044178b9f84aaba4cca33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7daa5af6c8044178b9f84aaba4cca33 already in progress: creating tablet
I20240503 02:12:21.043000 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet bc0e75639a4f4f168f9d833b090ae76a (table default.Table_5c5ea1ab22594d6b81fbd5b536231007 [id=9947c6f15d2444ca895eea2f96df51ae]) successfully deleted
I20240503 02:12:21.042951 5201 raft_consensus.cc:504] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:21.043251 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 894b753b71a04193905a190da320a260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 894b753b71a04193905a190da320a260 already in progress: creating tablet
I20240503 02:12:21.043731 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet a295609170a8482f8373dbf817b1eea1 (table default.Table_dde3e4c146264ea39a885cb77419db45 [id=07ea56ce8d3749dfbcbc1a79ea84693c]) successfully deleted
I20240503 02:12:21.044262 5201 leader_election.cc:288] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:21.044253 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:21.044893 4746 leader_election.cc:341] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.045253 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:21.045496 4529 raft_consensus.cc:3037] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.047113 5105 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.047571 5105 ts_tablet_manager.cc:1367] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.003s
I20240503 02:12:21.048282 4529 raft_consensus.cc:2454] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:21.049497 4746 leader_election.cc:302] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:21.050137 5201 raft_consensus.cc:2781] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.050995 5114 ts_tablet_manager.cc:1888] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:21.051162 5114 log.cc:1196] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/48135062d92949dc8386c43cf103ced4
I20240503 02:12:21.051823 5114 ts_tablet_manager.cc:1909] T 48135062d92949dc8386c43cf103ced4 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.053620 5114 tablet_replica.cc:330] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.053838 5114 raft_consensus.cc:2229] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.054055 5114 raft_consensus.cc:2258] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.054713 5114 ts_tablet_manager.cc:1875] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.055648 5105 raft_consensus.cc:348] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.056874 5105 raft_consensus.cc:374] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.057543 5105 raft_consensus.cc:729] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:21.058410 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22776c81d1a24d368abb68274cf32a0b consensus state has no leader
I20240503 02:12:21.058862 5105 consensus_queue.cc:260] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.061232 5201 raft_consensus.cc:686] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:21.061851 5201 consensus_queue.cc:237] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.062988 5114 ts_tablet_manager.cc:1888] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.063119 5114 log.cc:1196] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/effb2ce36a31477382865461d89917be
I20240503 02:12:21.063509 5114 ts_tablet_manager.cc:1909] T effb2ce36a31477382865461d89917be P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.066028 5114 tablet_replica.cc:330] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.066279 5114 raft_consensus.cc:2229] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.066468 5114 raft_consensus.cc:2258] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.067063 5114 ts_tablet_manager.cc:1875] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.073493 5276 raft_consensus.cc:482] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.073700 5276 raft_consensus.cc:504] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.074558 5276 leader_election.cc:288] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:21.076632 5114 ts_tablet_manager.cc:1888] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.076869 5114 log.cc:1196] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:21.077359 5114 ts_tablet_manager.cc:1909] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.078415 5114 tablet_replica.cc:330] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.078688 5114 raft_consensus.cc:2229] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.078965 5114 raft_consensus.cc:2258] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.079659 5114 ts_tablet_manager.cc:1875] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:21.080197 4746 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:21.081211 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2fd053276a0848d6bdf3010494bcaa0b" } modify_external_catalogs: true
I20240503 02:12:21.081496 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2fd053276a0848d6bdf3010494bcaa0b" } modify_external_catalogs: true
I20240503 02:12:21.082289 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_00ad2435ac2849cd8807b5bfd284c6e0" } new_table_name: "default.Table_29dccfb7e53045af9cc7c2f5c885f95b" modify_external_catalogs: true
I20240503 02:12:21.084832 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_34d04f474eb84a42948c4bde48e80439" } new_table_name: "default.Table_26db706b59c346f59cb5df08c11c1d22" modify_external_catalogs: true
I20240503 02:12:21.088164 5114 ts_tablet_manager.cc:1888] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.088325 5114 log.cc:1196] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/21e625e02b764ff2b7279b35dd10b08c
I20240503 02:12:21.088771 5114 ts_tablet_manager.cc:1909] T 21e625e02b764ff2b7279b35dd10b08c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.088673 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2193a952b28f4000b0472d610e708f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.092116 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b16c409173a84334895e0c40773793c3" } new_table_name: "default.Table_dce5dc7647cd4035b6cb74f11e45c718" modify_external_catalogs: true
I20240503 02:12:21.092594 5114 tablet_replica.cc:330] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.092864 5114 raft_consensus.cc:2229] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.092847 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a829c48a5c24d6d9e6329ac34151d1f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:21.093080 5114 raft_consensus.cc:2258] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.093312 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69392fa5f441049437639e143a7491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.093540 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet effb2ce36a31477382865461d89917be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:17 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.093789 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab (DEFAULT_TABLE table=default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.094005 5114 ts_tablet_manager.cc:1875] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.094214 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d (DEFAULT_TABLE table=default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.094566 5106 raft_consensus.cc:374] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.094666 5106 raft_consensus.cc:729] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.094936 5106 consensus_queue.cc:260] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.095504 5106 ts_tablet_manager.cc:1398] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.273s user 0.001s sys 0.001s
I20240503 02:12:21.095842 5106 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.096956 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.098377 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_37e1baa613ca4b10bb2727227e93163b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.099925 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f053537029a44d5a519b612d09d19a1 consensus state has no leader
W20240503 02:12:21.100855 4462 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.101080 4462 leader_election.cc:302] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:21.101188 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7394c9d374404fb199ab6ff170d7febe consensus state has no leader
W20240503 02:12:21.043980 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 387a596cfe654aa99e83d070b7884e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 387a596cfe654aa99e83d070b7884e6b already in progress: creating tablet
W20240503 02:12:21.101295 4462 raft_consensus.cc:2670] T 8a829c48a5c24d6d9e6329ac34151d1f P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:21.101614 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 57470.3us
I20240503 02:12:21.102751 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5a2971f0db2147bb875eba0e1a871f95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.104866 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:21.105182 4608 leader_election.cc:341] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 634beeb9d8394b1da8c4ad79ba7e26ac
I20240503 02:12:21.105242 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.105360 4608 leader_election.cc:302] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:21.105470 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:21.105634 4608 raft_consensus.cc:2670] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:21.105859 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:21.105887 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 96067.8us
I20240503 02:12:21.106361 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:21.106382 4608 leader_election.cc:341] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: effb2ce36a31477382865461d89917be
I20240503 02:12:21.106719 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53017352b8f34955b116bbfd32f04854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.107450 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d719c58974442281c7d694e1ca485e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:21.107635 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d919b9262ac48cbb8b3f893918b5590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.108103 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885129a4efc5475b97b7ac68e1db5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.108218 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a829c48a5c24d6d9e6329ac34151d1f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:21.109026 4746 leader_election.cc:341] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:21.109436 4746 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.109556 4746 leader_election.cc:302] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:21.110002 5276 raft_consensus.cc:2726] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:21.111804 4608 leader_election.cc:302] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:21.112430 5256 raft_consensus.cc:2781] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:21.112581 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e366ce9d79b3458fb26e96b6e9278fbe has no consensus state
I20240503 02:12:21.116046 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b72c20a4b5a4d2ea266f031733d1aab. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:21.116616 4608 leader_election.cc:341] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 7b69392fa5f441049437639e143a7491
I20240503 02:12:21.116660 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet e05d88a5c49b43bea6d46feccaaa6920 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:21.117122 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2193a952b28f4000b0472d610e708f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.119395 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37e1baa613ca4b10bb2727227e93163b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.104671 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b28bb9ac773c4bc685cb8193e574460b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b28bb9ac773c4bc685cb8193e574460b already in progress: creating tablet
I20240503 02:12:21.122433 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b161628065704c798020edbe098844c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.121335 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 48135062d92949dc8386c43cf103ced4 (table default.Table_8278654b5129427a9721eed448fc1758 [id=8b3bd9d6487e473089474aa5cc736cf4]) successfully deleted
W20240503 02:12:21.122011 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a2971f0db2147bb875eba0e1a871f95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.124248 5106 tablet_bootstrap.cc:654] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.121959 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cccdd46e80f149d58dbedf595291b449"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.124948 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cccdd46e80f149d58dbedf595291b449 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.126405 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4dac17dca3442d8934ac357779b78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.138954 5159 ts_tablet_manager.cc:1888] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.139259 5159 log.cc:1196] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:21.139792 5159 ts_tablet_manager.cc:1909] T 0f72aa1636df46ce9eec9c69c2f70c49 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.123287 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet effb2ce36a31477382865461d89917be (table default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]) successfully deleted
W20240503 02:12:21.143673 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 110c67212ec345028024da32968869ba because the tablet was not found. No further retry: Not found: Tablet not found: 110c67212ec345028024da32968869ba
W20240503 02:12:21.144024 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet bbab4368e73b4944b9853c456a4db7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbab4368e73b4944b9853c456a4db7bd already in progress: creating tablet
I20240503 02:12:21.146867 5103 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.147058 5103 ts_tablet_manager.cc:1367] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.203s user 0.003s sys 0.000s
I20240503 02:12:21.147888 5103 raft_consensus.cc:348] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.148135 5103 raft_consensus.cc:374] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.148234 5103 raft_consensus.cc:729] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.148492 5103 consensus_queue.cc:260] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.152220 5103 ts_tablet_manager.cc:1398] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:21.152630 5103 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.155746 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2f1e52a8b5d44129acbcd3c03132861e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.156442 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f1e52a8b5d44129acbcd3c03132861e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.157440 5103 tablet_bootstrap.cc:654] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.158279 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ed808d35bcc41bab738b11c2deca02c" } modify_external_catalogs: true
I20240503 02:12:21.158500 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ed808d35bcc41bab738b11c2deca02c" } modify_external_catalogs: true
W20240503 02:12:21.144358 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd402c0eb9e4cf49dc722f2dd214e09 already in progress: creating tablet
I20240503 02:12:21.162048 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 8a829c48a5c24d6d9e6329ac34151d1f (table default.Table_4ab6febbc9724cf78cc642abf953ee61 [id=cad57d8578d2442984a701a12ebd16ca]) successfully deleted
I20240503 02:12:21.162452 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 21e625e02b764ff2b7279b35dd10b08c (table default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]) successfully deleted
W20240503 02:12:21.162776 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 53dad4e45a184dc1ae4a82021a48e9be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53dad4e45a184dc1ae4a82021a48e9be already in progress: creating tablet
W20240503 02:12:21.163108 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d989197a5d6e42379df6d8add449a913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d989197a5d6e42379df6d8add449a913 already in progress: creating tablet
W20240503 02:12:21.163652 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 09005fb01c384122a7310c1eea12d3f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09005fb01c384122a7310c1eea12d3f0 already in progress: creating tablet
I20240503 02:12:21.168045 5103 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.168239 5103 ts_tablet_manager.cc:1367] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 02:12:21.169059 5103 raft_consensus.cc:348] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.169289 5103 raft_consensus.cc:374] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.169468 5103 raft_consensus.cc:729] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.169713 5103 consensus_queue.cc:260] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.170276 5103 ts_tablet_manager.cc:1398] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.172366 5103 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.174259 5103 tablet_bootstrap.cc:654] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.101294 5114 ts_tablet_manager.cc:1888] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.174647 5114 log.cc:1196] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:21.175068 5114 ts_tablet_manager.cc:1909] T 0f72aa1636df46ce9eec9c69c2f70c49 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:21.180857 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c already in progress: creating tablet
I20240503 02:12:21.181248 5115 ts_tablet_manager.cc:1888] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.181444 5115 log.cc:1196] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/634beeb9d8394b1da8c4ad79ba7e26ac
W20240503 02:12:21.181579 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 36a14c4f785f4514ae74b8a0b3073503 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36a14c4f785f4514ae74b8a0b3073503 already in progress: creating tablet
I20240503 02:12:21.181838 5115 ts_tablet_manager.cc:1909] T 634beeb9d8394b1da8c4ad79ba7e26ac P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.182440 5115 tablet_replica.cc:330] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.182655 5115 raft_consensus.cc:2229] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.182920 5115 raft_consensus.cc:2258] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.183849 5256 raft_consensus.cc:686] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:21.184428 5256 consensus_queue.cc:237] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:21.185957 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 67633.8us
W20240503 02:12:21.186287 4608 leader_election.cc:341] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 7b69392fa5f441049437639e143a7491
I20240503 02:12:21.186488 4608 leader_election.cc:302] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:21.188298 5280 raft_consensus.cc:2726] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:21.189478 5159 tablet_replica.cc:330] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:21.188047 5114 tablet_replica.cc:330] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.189716 5114 raft_consensus.cc:2229] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.189795 5159 raft_consensus.cc:2229] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.189949 5114 raft_consensus.cc:2258] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.190376 5159 raft_consensus.cc:2258] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.191234 5159 ts_tablet_manager.cc:1875] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.193637 5106 tablet_bootstrap.cc:492] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.193872 5106 ts_tablet_manager.cc:1367] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.005s
I20240503 02:12:21.194778 5106 raft_consensus.cc:348] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.195223 5106 raft_consensus.cc:374] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.195362 5115 ts_tablet_manager.cc:1875] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:21.195474 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 19fb0929bd234788b5313d96d2e4845c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fb0929bd234788b5313d96d2e4845c already in progress: creating tablet
I20240503 02:12:21.197101 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5c40f1f6b40308eed26b64d047494 (DEFAULT_TABLE table=default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.197600 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5c40f1f6b40308eed26b64d047494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.197919 5106 raft_consensus.cc:729] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.198199 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba" } modify_external_catalogs: true
I20240503 02:12:21.198454 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba" } modify_external_catalogs: true
I20240503 02:12:21.198833 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_464b5c2e54804303a2039f77c6a57dd4" } modify_external_catalogs: true
I20240503 02:12:21.199434 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_464b5c2e54804303a2039f77c6a57dd4" } modify_external_catalogs: true
I20240503 02:12:21.202927 5201 raft_consensus.cc:482] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.203169 5201 raft_consensus.cc:504] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.204497 5201 leader_election.cc:288] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:21.205247 4746 leader_election.cc:341] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 851756dc1a5942919c85967f83eca29a
I20240503 02:12:21.204022 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7fcef0fe6b204a57aef33be1149303db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.206797 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fcef0fe6b204a57aef33be1149303db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.209048 5114 ts_tablet_manager.cc:1875] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.211149 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851756dc1a5942919c85967f83eca29a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:21.211783 4746 leader_election.cc:341] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.211953 4746 leader_election.cc:302] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:21.212462 5201 raft_consensus.cc:2726] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:21.212800 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_631045fb785640fca1c307babc5c2276" } new_table_name: "default.Table_b5488cf4174e4180bace336add141933" modify_external_catalogs: true
I20240503 02:12:21.213831 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5c40f1f6b40308eed26b64d047494 (DEFAULT_TABLE table=default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.214113 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.214268 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5c40f1f6b40308eed26b64d047494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.214347 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.214478 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.214766 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.215912 5106 consensus_queue.cc:260] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.216754 5106 ts_tablet_manager.cc:1398] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 02:12:21.217118 5106 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.218549 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.219033 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.220449 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_26db706b59c346f59cb5df08c11c1d22" } modify_external_catalogs: true
I20240503 02:12:21.221627 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 (DEFAULT_TABLE table=default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.222067 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152f205cf1b4ec7ae03605bbfa007e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.223044 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 (DEFAULT_TABLE table=default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.223542 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152f205cf1b4ec7ae03605bbfa007e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.229274 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5c40f1f6b40308eed26b64d047494 (DEFAULT_TABLE table=default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.232481 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5c40f1f6b40308eed26b64d047494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.232508 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_26db706b59c346f59cb5df08c11c1d22" } modify_external_catalogs: true
I20240503 02:12:21.235253 4507 tablet_service.cc:1467] Processing CreateTablet for tablet f2f9b962c9264f87b798df33544cb15d (DEFAULT_TABLE table=default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.235463 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 17c2a8190210439fb7589cb02ec82bc2 (DEFAULT_TABLE table=default.Table_150d9a5a76ef45ba89eae071bd04eab6 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.235672 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 (DEFAULT_TABLE table=default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.235874 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c2a8190210439fb7589cb02ec82bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.236055 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 17c2a8190210439fb7589cb02ec82bc2 (DEFAULT_TABLE table=default.Table_150d9a5a76ef45ba89eae071bd04eab6 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.236200 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 (DEFAULT_TABLE table=default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.236280 5106 tablet_bootstrap.cc:654] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.236577 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7394c9d374404fb199ab6ff170d7febe (DEFAULT_TABLE table=default.Table_00ad2435ac2849cd8807b5bfd284c6e0 [id=646347328e89442b8713fa69f5078052]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.236768 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 (DEFAULT_TABLE table=default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.237205 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e9faaa6a584ec38b9da9d2670e47f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.235756 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b962c9264f87b798df33544cb15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.235410 4782 tablet_service.cc:1467] Processing CreateTablet for tablet f2f9b962c9264f87b798df33544cb15d (DEFAULT_TABLE table=default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.241294 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e9faaa6a584ec38b9da9d2670e47f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.242383 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 17c2a8190210439fb7589cb02ec82bc2 (DEFAULT_TABLE table=default.Table_150d9a5a76ef45ba89eae071bd04eab6 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.242388 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 7394c9d374404fb199ab6ff170d7febe (DEFAULT_TABLE table=default.Table_00ad2435ac2849cd8807b5bfd284c6e0 [id=646347328e89442b8713fa69f5078052]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.242974 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c2a8190210439fb7589cb02ec82bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.243221 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c2a8190210439fb7589cb02ec82bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.243214 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394c9d374404fb199ab6ff170d7febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.250006 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 (DEFAULT_TABLE table=default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.250607 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 0f72aa1636df46ce9eec9c69c2f70c49 (table default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]) successfully deleted
I20240503 02:12:21.251013 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 7394c9d374404fb199ab6ff170d7febe (DEFAULT_TABLE table=default.Table_00ad2435ac2849cd8807b5bfd284c6e0 [id=646347328e89442b8713fa69f5078052]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.251528 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394c9d374404fb199ab6ff170d7febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.252589 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b962c9264f87b798df33544cb15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.252885 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e9faaa6a584ec38b9da9d2670e47f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.252374 4646 tablet_service.cc:1467] Processing CreateTablet for tablet f2f9b962c9264f87b798df33544cb15d (DEFAULT_TABLE table=default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.253602 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 29e5c40f1f6b40308eed26b64d047494
I20240503 02:12:21.253767 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b962c9264f87b798df33544cb15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.254521 5115 ts_tablet_manager.cc:1888] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.254654 5115 log.cc:1196] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c46ea1084304449e895be230d86471a9
I20240503 02:12:21.257395 4273 catalog_manager.cc:5484] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.261828 5115 ts_tablet_manager.cc:1909] T c46ea1084304449e895be230d86471a9 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.262436 5115 tablet_replica.cc:330] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.263336 5115 raft_consensus.cc:2229] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.267763 5243 raft_consensus.cc:482] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.268013 5243 raft_consensus.cc:504] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.268885 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152f205cf1b4ec7ae03605bbfa007e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.269024 5243 leader_election.cc:288] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:21.270313 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7daa5af6c8044178b9f84aaba4cca33" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:21.271016 4462 leader_election.cc:341] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.273792 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394c9d374404fb199ab6ff170d7febe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.273988 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7daa5af6c8044178b9f84aaba4cca33" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:21.274539 4462 leader_election.cc:341] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.274696 4462 leader_election.cc:302] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:21.275373 5286 raft_consensus.cc:2726] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:21.275560 5115 raft_consensus.cc:2258] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.276268 5115 ts_tablet_manager.cc:1875] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.277473 5105 ts_tablet_manager.cc:1398] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.230s user 0.030s sys 0.038s
I20240503 02:12:21.281735 5106 tablet_bootstrap.cc:492] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:21.285499 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 53dad4e45a184dc1ae4a82021a48e9be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53dad4e45a184dc1ae4a82021a48e9be already in progress: creating tablet
I20240503 02:12:21.285435 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1d1adcf60a414b9b8d0d7621be5c21c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.285854 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d989197a5d6e42379df6d8add449a913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d989197a5d6e42379df6d8add449a913 already in progress: creating tablet
W20240503 02:12:21.286123 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 617ad14927c1441b800a204aefc2e619 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617ad14927c1441b800a204aefc2e619 already in progress: creating tablet
W20240503 02:12:21.286375 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 09005fb01c384122a7310c1eea12d3f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09005fb01c384122a7310c1eea12d3f0 already in progress: creating tablet
W20240503 02:12:21.286590 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c already in progress: creating tablet
W20240503 02:12:21.286794 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 36a14c4f785f4514ae74b8a0b3073503 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36a14c4f785f4514ae74b8a0b3073503 already in progress: creating tablet
W20240503 02:12:21.286945 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e94031465ce7407dab0f90ad8809ac21 consensus state has no leader
W20240503 02:12:21.287024 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 19fb0929bd234788b5313d96d2e4845c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fb0929bd234788b5313d96d2e4845c already in progress: creating tablet
I20240503 02:12:21.287756 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5f56701ed6424c60bf1a985eea1936fe
W20240503 02:12:21.288178 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f22a4b8015354db493652d8f336a1265 because the tablet was not found. No further retry: Not found: Tablet not found: f22a4b8015354db493652d8f336a1265
I20240503 02:12:21.286010 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_72645b71fd5b41039586175c6982ccec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.288775 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet bbab4368e73b4944b9853c456a4db7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbab4368e73b4944b9853c456a4db7bd already in progress: creating tablet
W20240503 02:12:21.288951 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72645b71fd5b41039586175c6982ccec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.289024 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd402c0eb9e4cf49dc722f2dd214e09 already in progress: creating tablet
W20240503 02:12:21.286132 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.290755 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 634beeb9d8394b1da8c4ad79ba7e26ac (table default.Table_4227e90304ee491ca298dabd17aa7b44 [id=536a3d22f6e34248b7936ba2e7a05fd5]) successfully deleted
I20240503 02:12:21.291379 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c46ea1084304449e895be230d86471a9 (table default.Table_94169f163e904089be031c1def8b5ba6 [id=a9a1676f5f6a41fb9d7614f6c9796791]) successfully deleted
I20240503 02:12:21.293737 5159 ts_tablet_manager.cc:1888] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.293907 5159 log.cc:1196] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:21.293954 5106 ts_tablet_manager.cc:1367] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.004s
I20240503 02:12:21.294306 5159 ts_tablet_manager.cc:1909] T 8a829c48a5c24d6d9e6329ac34151d1f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.294818 5106 raft_consensus.cc:348] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.295058 5106 raft_consensus.cc:374] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.295151 5106 raft_consensus.cc:729] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.295466 5106 consensus_queue.cc:260] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.296015 5106 ts_tablet_manager.cc:1398] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:21.296346 5106 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.298883 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_19707a170e734efab69018ce4a0d61f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.299536 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19707a170e734efab69018ce4a0d61f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.301542 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c00f7cc799d34540b77a581d4a9a4561" } new_table_name: "default.Table_100da0be301548829b543438a8d63e02" modify_external_catalogs: true
I20240503 02:12:21.304281 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet e05d88a5c49b43bea6d46feccaaa6920
I20240503 02:12:21.304771 4271 catalog_manager.cc:6554] ReplaceTablet: tablet e05d88a5c49b43bea6d46feccaaa6920 deleted and replaced by tablet 59855ab3e42b4e019b769007ccbbc360
I20240503 02:12:21.305069 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet c46ea1084304449e895be230d86471a9 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.305369 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 0f72aa1636df46ce9eec9c69c2f70c49 (table default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]) successfully deleted
I20240503 02:12:21.305423 5106 tablet_bootstrap.cc:654] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:21.305637 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 617ad14927c1441b800a204aefc2e619 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617ad14927c1441b800a204aefc2e619 already in progress: creating tablet
I20240503 02:12:21.306288 5159 tablet_replica.cc:330] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:21.306496 5159 raft_consensus.cc:2229] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.306692 5159 raft_consensus.cc:2258] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.307878 5115 ts_tablet_manager.cc:1888] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.308135 5115 log.cc:1196] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/e0fee5bceac941cd83471e5cad5dd463
I20240503 02:12:21.308764 5115 ts_tablet_manager.cc:1909] T e0fee5bceac941cd83471e5cad5dd463 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.308615 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0fbcd3f402c1447f86400a7052a931a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.309278 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fbcd3f402c1447f86400a7052a931a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.309922 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3c6e5e7d40249b112144975ba220 (DEFAULT_TABLE table=default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.310431 5115 tablet_replica.cc:330] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.310721 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ef3c6e5e7d40249b112144975ba220. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.311488 5115 raft_consensus.cc:2229] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.311996 5115 raft_consensus.cc:2258] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.313899 5115 ts_tablet_manager.cc:1875] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.315445 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_070e3cfe611e43baaec51dd9b0c1d7e9" } new_table_name: "default.Table_eae0f8da0ab0497e8731eb30c28c8c7c" modify_external_catalogs: true
I20240503 02:12:21.317749 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3c6e5e7d40249b112144975ba220 (DEFAULT_TABLE table=default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.318181 5105 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.318210 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 7461aa76a1af41cb9253285bda708f30 (DEFAULT_TABLE table=default.Table_3f5c5ee57b8c4f27987de424a90281cb [id=668afa5a4e424ea5b5398d690a79fa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.318464 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ef3c6e5e7d40249b112144975ba220. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.318675 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7461aa76a1af41cb9253285bda708f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.319031 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3c6e5e7d40249b112144975ba220 (DEFAULT_TABLE table=default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.319514 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ef3c6e5e7d40249b112144975ba220. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.320149 5105 tablet_bootstrap.cc:654] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.321866 5159 ts_tablet_manager.cc:1875] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.326300 5106 tablet_bootstrap.cc:492] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.326534 5106 ts_tablet_manager.cc:1367] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20240503 02:12:21.327332 5106 raft_consensus.cc:348] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.327636 5106 raft_consensus.cc:374] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.327776 5106 raft_consensus.cc:729] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.328083 5106 consensus_queue.cc:260] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.328722 5106 ts_tablet_manager.cc:1398] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:21.331996 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2f053537029a44d5a519b612d09d19a1
I20240503 02:12:21.332427 5106 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.334266 5106 tablet_bootstrap.cc:654] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.335882 5105 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.336074 5105 ts_tablet_manager.cc:1367] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240503 02:12:21.336678 5115 ts_tablet_manager.cc:1888] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.337085 5115 log.cc:1196] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7b69392fa5f441049437639e143a7491
I20240503 02:12:21.337044 5105 raft_consensus.cc:348] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.337175 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b5488cf4174e4180bace336add141933" } new_table_name: "default.Table_6c6c926459ab48ef88b1da1b7358dbdd" modify_external_catalogs: true
I20240503 02:12:21.337308 5105 raft_consensus.cc:374] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.337428 5105 raft_consensus.cc:729] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.337706 5105 consensus_queue.cc:260] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.338235 5115 ts_tablet_manager.cc:1909] T 7b69392fa5f441049437639e143a7491 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.338243 5105 ts_tablet_manager.cc:1398] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.340188 5115 tablet_replica.cc:330] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.340338 5159 ts_tablet_manager.cc:1888] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.340492 5159 log.cc:1196] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/851756dc1a5942919c85967f83eca29a
I20240503 02:12:21.340747 5115 raft_consensus.cc:2229] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:21.340853 5159 ts_tablet_manager.cc:1909] T 851756dc1a5942919c85967f83eca29a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.341312 5115 pending_rounds.cc:62] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:21.341734 5115 pending_rounds.cc:68] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420789495930880 op_type: NO_OP noop_request { }
I20240503 02:12:21.342185 5115 raft_consensus.cc:2866] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:21.342562 5115 raft_consensus.cc:2258] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.343663 5115 ts_tablet_manager.cc:1875] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.344790 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.345098 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.345571 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.346103 5105 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.346555 5103 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.346752 5103 ts_tablet_manager.cc:1367] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.174s user 0.013s sys 0.018s
I20240503 02:12:21.346520 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.346915 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 7461aa76a1af41cb9253285bda708f30 (DEFAULT_TABLE table=default.Table_3f5c5ee57b8c4f27987de424a90281cb [id=668afa5a4e424ea5b5398d690a79fa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.347618 5103 raft_consensus.cc:348] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.347844 5103 raft_consensus.cc:374] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.347854 5105 tablet_bootstrap.cc:654] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.347910 5103 raft_consensus.cc:729] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.347929 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7461aa76a1af41cb9253285bda708f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.348184 5103 consensus_queue.cc:260] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.349826 4654 tablet_service.cc:1467] Processing CreateTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 (DEFAULT_TABLE table=default.Table_827711fb4f7843b88f3481b51b2ee727 [id=65c98763e63d4447b90a16f448bccffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.350106 4785 tablet_service.cc:1467] Processing CreateTablet for tablet e366ce9d79b3458fb26e96b6e9278fbe (DEFAULT_TABLE table=default.Table_dce5dc7647cd4035b6cb74f11e45c718 [id=7dd3df315372418f9c30f7a0e506fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.350260 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91abaf51abb4c2ca65412c3771c1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.350517 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e366ce9d79b3458fb26e96b6e9278fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.351303 5103 ts_tablet_manager.cc:1398] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:21.351627 5103 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.357198 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 885129a4efc5475b97b7ac68e1db5e1d
I20240503 02:12:21.357817 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b69392fa5f441049437639e143a7491 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.358439 4792 tablet_service.cc:1467] Processing CreateTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 (DEFAULT_TABLE table=default.Table_827711fb4f7843b88f3481b51b2ee727 [id=65c98763e63d4447b90a16f448bccffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.359473 4503 tablet_service.cc:1467] Processing CreateTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 (DEFAULT_TABLE table=default.Table_827711fb4f7843b88f3481b51b2ee727 [id=65c98763e63d4447b90a16f448bccffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.359921 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91abaf51abb4c2ca65412c3771c1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.361812 5103 tablet_bootstrap.cc:654] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.362347 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91abaf51abb4c2ca65412c3771c1ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.363276 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_827711fb4f7843b88f3481b51b2ee727" } new_table_name: "default.Table_df633d54f27d4e71b052cbf0582d8ba2" modify_external_catalogs: true
I20240503 02:12:21.366240 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 78b21a3010a4470288fbdd684ef64e77 (DEFAULT_TABLE table=default.Table_31852ddf01b44f4a88e80b4216b590c9 [id=5d52bb117988409d974ea961f9f7bab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.366721 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b21a3010a4470288fbdd684ef64e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.368893 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 7461aa76a1af41cb9253285bda708f30 (DEFAULT_TABLE table=default.Table_3f5c5ee57b8c4f27987de424a90281cb [id=668afa5a4e424ea5b5398d690a79fa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.370424 4493 tablet_service.cc:1467] Processing CreateTablet for tablet e366ce9d79b3458fb26e96b6e9278fbe (DEFAULT_TABLE table=default.Table_dce5dc7647cd4035b6cb74f11e45c718 [id=7dd3df315372418f9c30f7a0e506fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.371816 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 53dad4e45a184dc1ae4a82021a48e9be from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.372861 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7461aa76a1af41cb9253285bda708f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.373294 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e366ce9d79b3458fb26e96b6e9278fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.373437 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 78b21a3010a4470288fbdd684ef64e77 (DEFAULT_TABLE table=default.Table_31852ddf01b44f4a88e80b4216b590c9 [id=5d52bb117988409d974ea961f9f7bab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.373816 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b21a3010a4470288fbdd684ef64e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.375790 5114 ts_tablet_manager.cc:1888] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.366242 4654 tablet_service.cc:1467] Processing CreateTablet for tablet e366ce9d79b3458fb26e96b6e9278fbe (DEFAULT_TABLE table=default.Table_dce5dc7647cd4035b6cb74f11e45c718 [id=7dd3df315372418f9c30f7a0e506fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.378340 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e366ce9d79b3458fb26e96b6e9278fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.378495 5114 log.cc:1196] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/ab3c55f80a644cefa87b5769f15bb7f0
I20240503 02:12:21.379338 5114 ts_tablet_manager.cc:1909] T ab3c55f80a644cefa87b5769f15bb7f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.382941 5115 ts_tablet_manager.cc:1888] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:21.384817 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 53dad4e45a184dc1ae4a82021a48e9be from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.386080 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3c3549c20fa149b78d707a1a111895d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.386505 5114 tablet_replica.cc:330] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:21.386719 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c3549c20fa149b78d707a1a111895d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.386726 5114 raft_consensus.cc:2229] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.386906 5114 raft_consensus.cc:2258] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.387159 5115 log.cc:1196] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/effb2ce36a31477382865461d89917be
I20240503 02:12:21.387871 5115 ts_tablet_manager.cc:1909] T effb2ce36a31477382865461d89917be P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.389974 4500 tablet_service.cc:1467] Processing CreateTablet for tablet ea7afb429d64497abcd9f1d0910651c3 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.389971 4637 tablet_service.cc:1467] Processing CreateTablet for tablet ea7afb429d64497abcd9f1d0910651c3 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.390520 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7afb429d64497abcd9f1d0910651c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.390656 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7afb429d64497abcd9f1d0910651c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.393362 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.393333 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 78b21a3010a4470288fbdd684ef64e77 (DEFAULT_TABLE table=default.Table_31852ddf01b44f4a88e80b4216b590c9 [id=5d52bb117988409d974ea961f9f7bab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.394482 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.394517 4782 tablet_service.cc:1467] Processing CreateTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc (DEFAULT_TABLE table=default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.394954 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74f4816cad04a9eb5934aaba86c2fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.394922 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.395365 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.395721 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65774b69a8bd4664a6b55b84eba37f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.396209 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet c126b8ef15c5432abf037e28a5eeba58 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d5e627d818245e0bdeb16531499f7d5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.396790 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_308844f35cb448378721f1da35d38c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.397217 4784 tablet_service.cc:1467] Processing CreateTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 (DEFAULT_TABLE table=default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5 [id=64f4026243214e1bbed391c6e0e7df76]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:21.397349 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_308844f35cb448378721f1da35d38c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.397593 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f146f7cce0b24d3087bfe0973e2b7ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.400601 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.401224 4647 tablet_service.cc:1467] Processing CreateTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 (DEFAULT_TABLE table=default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5 [id=64f4026243214e1bbed391c6e0e7df76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.401417 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 41d44fe8f6334572996665bdec60e2e5 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.401574 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 41d44fe8f6334572996665bdec60e2e5 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.401784 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d44fe8f6334572996665bdec60e2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.401795 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (replaced by tablet 59855ab3e42b4e019b769007ccbbc360) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.402992 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 41d44fe8f6334572996665bdec60e2e5 (DEFAULT_TABLE table=default.Table_0414bb3004e94f78a6ff888c2ef0cb8b [id=d6973688027a4da2ba16399151d8bedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.403369 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d44fe8f6334572996665bdec60e2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.405285 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (replaced by tablet 59855ab3e42b4e019b769007ccbbc360) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.405305 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f (DEFAULT_TABLE table=default.Table_033290f38d39444aad4953c5f91ffaeb [id=522166873101443c80e2c2159f181c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.405681 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65774b69a8bd4664a6b55b84eba37f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.406026 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (replaced by tablet 59855ab3e42b4e019b769007ccbbc360) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.406397 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_143457686a2e414c8d11b68e09860a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.406940 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_143457686a2e414c8d11b68e09860a08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.407781 4646 tablet_service.cc:1467] Processing CreateTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc (DEFAULT_TABLE table=default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.409543 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8a829c48a5c24d6d9e6329ac34151d1f (table default.Table_4ab6febbc9724cf78cc642abf953ee61 [id=cad57d8578d2442984a701a12ebd16ca]) successfully deleted
I20240503 02:12:21.410184 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 851756dc1a5942919c85967f83eca29a (table default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]) successfully deleted
I20240503 02:12:21.410692 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet e0fee5bceac941cd83471e5cad5dd463 (table default.Table_73fd503758ae407593561d3ad8412920 [id=2ae28293f5d94b5e897e57a2ce49c0bd]) successfully deleted
W20240503 02:12:21.410915 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8a829c48a5c24d6d9e6329ac34151d1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a829c48a5c24d6d9e6329ac34151d1f already in progress: creating tablet
W20240503 02:12:21.411150 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 19fb0929bd234788b5313d96d2e4845c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19fb0929bd234788b5313d96d2e4845c already in progress: creating tablet
I20240503 02:12:21.411432 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7b69392fa5f441049437639e143a7491 (table default.Table_71f9f712282348f1b44f2b37d9962c96 [id=77385b54332241e5af62e7755aba9596]) successfully deleted
I20240503 02:12:21.411890 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet effb2ce36a31477382865461d89917be (table default.Table_9e283dbc214c46a28983ff2cb567ee77 [id=3f0fb07609f24f5dbc1d31fbbf9b5000]) successfully deleted
W20240503 02:12:21.412113 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 53dad4e45a184dc1ae4a82021a48e9be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53dad4e45a184dc1ae4a82021a48e9be already in progress: creating tablet
W20240503 02:12:21.412341 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d989197a5d6e42379df6d8add449a913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d989197a5d6e42379df6d8add449a913 already in progress: creating tablet
W20240503 02:12:21.412565 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 617ad14927c1441b800a204aefc2e619 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617ad14927c1441b800a204aefc2e619 already in progress: creating tablet
W20240503 02:12:21.412786 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 09005fb01c384122a7310c1eea12d3f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09005fb01c384122a7310c1eea12d3f0 already in progress: creating tablet
I20240503 02:12:21.413012 4504 tablet_service.cc:1467] Processing CreateTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc (DEFAULT_TABLE table=default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.413511 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74f4816cad04a9eb5934aaba86c2fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.412750 4792 tablet_service.cc:1467] Processing CreateTablet for tablet ea7afb429d64497abcd9f1d0910651c3 (DEFAULT_TABLE table=default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.414417 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7afb429d64497abcd9f1d0910651c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.415726 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c4e56102ddd84ad58342dbdfb695e622"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.416270 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e56102ddd84ad58342dbdfb695e622 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.416733 5105 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.416970 5105 ts_tablet_manager.cc:1367] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.001s
I20240503 02:12:21.417825 5105 raft_consensus.cc:348] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.418143 5105 raft_consensus.cc:374] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.418295 5105 raft_consensus.cc:729] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.418534 4494 tablet_service.cc:1467] Processing CreateTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 (DEFAULT_TABLE table=default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5 [id=64f4026243214e1bbed391c6e0e7df76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.418598 5105 consensus_queue.cc:260] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.419023 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f146f7cce0b24d3087bfe0973e2b7ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.420413 5105 ts_tablet_manager.cc:1398] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:21.420992 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f146f7cce0b24d3087bfe0973e2b7ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.421310 5105 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:21.421352 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34eb69d5f4bd427a8e430a00fb076f83 consensus state has no leader
I20240503 02:12:21.423319 5106 tablet_bootstrap.cc:492] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.423492 5106 ts_tablet_manager.cc:1367] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.005s
I20240503 02:12:21.424228 5106 raft_consensus.cc:348] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.424639 5106 raft_consensus.cc:374] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.424728 5106 raft_consensus.cc:729] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.425017 5106 consensus_queue.cc:260] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.425273 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74f4816cad04a9eb5934aaba86c2fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.426506 5106 ts_tablet_manager.cc:1398] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:21.426843 5106 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.429172 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d44fe8f6334572996665bdec60e2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.430047 5106 tablet_bootstrap.cc:654] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.431747 4266 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 59855ab3e42b4e019b769007ccbbc360
W20240503 02:12:21.432456 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c already in progress: creating tablet
W20240503 02:12:21.432740 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 36a14c4f785f4514ae74b8a0b3073503 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36a14c4f785f4514ae74b8a0b3073503 already in progress: creating tablet
I20240503 02:12:21.433254 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65774b69a8bd4664a6b55b84eba37f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.433574 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.437589 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.438851 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet ab3c55f80a644cefa87b5769f15bb7f0 (table default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]) successfully deleted
W20240503 02:12:21.439615 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 29e5c40f1f6b40308eed26b64d047494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e5c40f1f6b40308eed26b64d047494 already in progress: creating tablet
W20240503 02:12:21.439864 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5f56701ed6424c60bf1a985eea1936fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f56701ed6424c60bf1a985eea1936fe already in progress: creating tablet
I20240503 02:12:21.444027 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.444360 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.446841 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b21a3010a4470288fbdd684ef64e77. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:21.447350 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f40706400f114def9e14e6d1d7114962 consensus state has no leader
I20240503 02:12:21.447055 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.448954 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.449276 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.450379 5114 ts_tablet_manager.cc:1875] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.450575 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fd4fb88436498999b5f339052ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.451323 5159 tablet_replica.cc:330] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:21.452941 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46560ac6d94943d2a8fb3b3904f2b5b6 consensus state has no leader
I20240503 02:12:21.453377 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.453594 5103 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.453701 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.453750 5103 ts_tablet_manager.cc:1367] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.000s
I20240503 02:12:21.454882 5159 raft_consensus.cc:2229] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.455116 5159 raft_consensus.cc:2258] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.456311 5105 tablet_bootstrap.cc:654] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.456627 5159 ts_tablet_manager.cc:1875] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.457614 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6f061523dfc54120ae18f90ddd6c06f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.458245 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f061523dfc54120ae18f90ddd6c06f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.458461 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b91abaf51abb4c2ca65412c3771c1ef2 consensus state has no leader
W20240503 02:12:21.460602 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e94031465ce7407dab0f90ad8809ac21 consensus state has no leader
I20240503 02:12:21.462849 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_422f8213fa2e4d269fc4499098056f64" } modify_external_catalogs: true
I20240503 02:12:21.463076 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_422f8213fa2e4d269fc4499098056f64" } modify_external_catalogs: true
I20240503 02:12:21.465471 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 110c67212ec345028024da32968869ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.472003 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet b487131ac3a842009b3fe54578ea091d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.472332 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet fc887bc4db1c4e268c4c9f23096d704c with delete_type TABLET_DATA_DELETED (replaced by tablet 894b753b71a04193905a190da320a260) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.473004 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0e75639a4f4f168f9d833b090ae76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.473717 5115 tablet_replica.cc:330] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:21.473814 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 29e5c40f1f6b40308eed26b64d047494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e5c40f1f6b40308eed26b64d047494 already in progress: creating tablet
W20240503 02:12:21.474105 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5f56701ed6424c60bf1a985eea1936fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f56701ed6424c60bf1a985eea1936fe already in progress: creating tablet
I20240503 02:12:21.474336 5115 raft_consensus.cc:2229] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.474555 5115 raft_consensus.cc:2258] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.454576 5103 raft_consensus.cc:348] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.475606 5103 raft_consensus.cc:374] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.475731 5103 raft_consensus.cc:729] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.475733 5115 ts_tablet_manager.cc:1875] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.476012 5103 consensus_queue.cc:260] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.479535 5103 ts_tablet_manager.cc:1398] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:12:21.484964 5106 tablet_bootstrap.cc:492] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.485750 5106 ts_tablet_manager.cc:1367] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.007s
I20240503 02:12:21.486925 5106 raft_consensus.cc:348] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.487227 5106 raft_consensus.cc:374] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.487354 5106 raft_consensus.cc:729] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.487661 5106 consensus_queue.cc:260] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.489004 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e6aa097db3d14a28992284c5876aba0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.489694 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6aa097db3d14a28992284c5876aba0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.491575 5103 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.492026 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_29dccfb7e53045af9cc7c2f5c885f95b" } new_table_name: "default.Table_be53cf9a5c3a418d9dc24120c17d3a4b" modify_external_catalogs: true
I20240503 02:12:21.492146 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_535d1456ce3640c6a7e33dbc4b2c305c" } modify_external_catalogs: true
I20240503 02:12:21.492365 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_535d1456ce3640c6a7e33dbc4b2c305c" } modify_external_catalogs: true
I20240503 02:12:21.494014 5103 tablet_bootstrap.cc:654] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.496937 5106 ts_tablet_manager.cc:1398] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:12:21.489156 5115 ts_tablet_manager.cc:1888] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.497664 5115 log.cc:1196] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c126b8ef15c5432abf037e28a5eeba58
I20240503 02:12:21.498144 5115 ts_tablet_manager.cc:1909] T c126b8ef15c5432abf037e28a5eeba58 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.499624 5115 tablet_replica.cc:330] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.499899 5115 raft_consensus.cc:2229] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.500129 5115 raft_consensus.cc:2258] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.497514 5106 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.500883 5159 ts_tablet_manager.cc:1888] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.501084 5159 log.cc:1196] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fe5117d4d59b4f0b87c32e8b27b085d9
I20240503 02:12:21.501446 5159 ts_tablet_manager.cc:1909] T fe5117d4d59b4f0b87c32e8b27b085d9 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.501621 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c126b8ef15c5432abf037e28a5eeba58 (table default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]) successfully deleted
W20240503 02:12:21.502004 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2f053537029a44d5a519b612d09d19a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f053537029a44d5a519b612d09d19a1 already in progress: creating tablet
W20240503 02:12:21.502288 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 885129a4efc5475b97b7ac68e1db5e1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885129a4efc5475b97b7ac68e1db5e1d already in progress: creating tablet
I20240503 02:12:21.502774 5106 tablet_bootstrap.cc:654] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.509663 5114 ts_tablet_manager.cc:1888] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.509968 5114 log.cc:1196] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/fe5117d4d59b4f0b87c32e8b27b085d9
I20240503 02:12:21.511026 5115 ts_tablet_manager.cc:1875] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.512365 5114 ts_tablet_manager.cc:1909] T fe5117d4d59b4f0b87c32e8b27b085d9 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.513667 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet fe5117d4d59b4f0b87c32e8b27b085d9 (table default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]) successfully deleted
W20240503 02:12:21.514336 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 29e5c40f1f6b40308eed26b64d047494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e5c40f1f6b40308eed26b64d047494 already in progress: creating tablet
W20240503 02:12:21.516355 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5f56701ed6424c60bf1a985eea1936fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f56701ed6424c60bf1a985eea1936fe already in progress: creating tablet
W20240503 02:12:21.518030 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 885129a4efc5475b97b7ac68e1db5e1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885129a4efc5475b97b7ac68e1db5e1d already in progress: creating tablet
W20240503 02:12:21.518311 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2f053537029a44d5a519b612d09d19a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f053537029a44d5a519b612d09d19a1 already in progress: creating tablet
I20240503 02:12:21.518563 5114 tablet_replica.cc:330] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.518805 5114 raft_consensus.cc:2229] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.518841 5106 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.519040 5106 ts_tablet_manager.cc:1367] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240503 02:12:21.519471 5114 raft_consensus.cc:2258] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.519996 5106 raft_consensus.cc:348] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.520362 5106 raft_consensus.cc:374] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.520509 5106 raft_consensus.cc:729] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.520850 5106 consensus_queue.cc:260] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.522358 5106 ts_tablet_manager.cc:1398] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:21.528306 5115 ts_tablet_manager.cc:1888] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.534662 5106 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.546742 5106 tablet_bootstrap.cc:654] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.547924 5115 log.cc:1196] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/110c67212ec345028024da32968869ba
I20240503 02:12:21.548568 5115 ts_tablet_manager.cc:1909] T 110c67212ec345028024da32968869ba P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.550159 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 110c67212ec345028024da32968869ba (table default.Table_0fc326172fb94ff3957ddea1033b4a7a [id=660f0ce51745452589c79fabe77db7c9]) successfully deleted
I20240503 02:12:21.551065 5115 tablet_replica.cc:330] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.554139 5115 raft_consensus.cc:2229] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.554858 5115 raft_consensus.cc:2258] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.555976 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 09005fb01c384122a7310c1eea12d3f0 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.556579 5106 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.556653 5115 ts_tablet_manager.cc:1875] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.556744 5106 ts_tablet_manager.cc:1367] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20240503 02:12:21.557606 5106 raft_consensus.cc:348] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.557801 5106 raft_consensus.cc:374] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.557893 5106 raft_consensus.cc:729] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.558250 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d5e627d818245e0bdeb16531499f7d5 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.558123 5106 consensus_queue.cc:260] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.558688 5106 ts_tablet_manager.cc:1398] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.558866 4503 tablet_service.cc:1467] Processing CreateTablet for tablet b28f66fe53064a46a0b98b78aa77115c (DEFAULT_TABLE table=default.Table_2f1e52a8b5d44129acbcd3c03132861e [id=2ad7c9ff8c2d4cb19b38ebf6008349d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.559016 5106 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.559338 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28f66fe53064a46a0b98b78aa77115c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.560609 4647 tablet_service.cc:1467] Processing CreateTablet for tablet b28f66fe53064a46a0b98b78aa77115c (DEFAULT_TABLE table=default.Table_2f1e52a8b5d44129acbcd3c03132861e [id=2ad7c9ff8c2d4cb19b38ebf6008349d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.561012 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28f66fe53064a46a0b98b78aa77115c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.562146 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:21.562345 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 7178760c21904207b6c8317093871f63 (DEFAULT_TABLE table=default.Table_5a2971f0db2147bb875eba0e1a871f95 [id=07e16e4056424605a70ccfaa25a24613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.562757 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178760c21904207b6c8317093871f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.563370 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:21.565307 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 7178760c21904207b6c8317093871f63 (DEFAULT_TABLE table=default.Table_5a2971f0db2147bb875eba0e1a871f95 [id=07e16e4056424605a70ccfaa25a24613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.565483 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 95681e67db1e4485876d530133e8a514 (DEFAULT_TABLE table=default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.565701 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178760c21904207b6c8317093871f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.565840 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95681e67db1e4485876d530133e8a514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.569208 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 48e713ee37d042309166b0f118e14986 (DEFAULT_TABLE table=default.Table_37e1baa613ca4b10bb2727227e93163b [id=b4c6a45a47e647b1947d0d7f2ea30afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.569245 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 95681e67db1e4485876d530133e8a514 (DEFAULT_TABLE table=default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.569597 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95681e67db1e4485876d530133e8a514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.569614 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e713ee37d042309166b0f118e14986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.572088 4787 tablet_service.cc:1467] Processing CreateTablet for tablet d39f042afb954daa923502ef12377bd5 (DEFAULT_TABLE table=default.Table_cccdd46e80f149d58dbedf595291b449 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.572510 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f042afb954daa923502ef12377bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.573228 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 48e713ee37d042309166b0f118e14986 (DEFAULT_TABLE table=default.Table_37e1baa613ca4b10bb2727227e93163b [id=b4c6a45a47e647b1947d0d7f2ea30afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.573470 4500 tablet_service.cc:1467] Processing CreateTablet for tablet d39f042afb954daa923502ef12377bd5 (DEFAULT_TABLE table=default.Table_cccdd46e80f149d58dbedf595291b449 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.573601 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e713ee37d042309166b0f118e14986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.573804 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f042afb954daa923502ef12377bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.577214 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94fb9d53044d796e2973143d888fb (DEFAULT_TABLE table=default.Table_7fcef0fe6b204a57aef33be1149303db [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.577596 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94fb9d53044d796e2973143d888fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.578641 5115 ts_tablet_manager.cc:1888] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.578797 5115 log.cc:1196] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b487131ac3a842009b3fe54578ea091d
I20240503 02:12:21.579182 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 59855ab3e42b4e019b769007ccbbc360 (DEFAULT_TABLE table=default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.579471 4654 tablet_service.cc:1467] Processing CreateTablet for tablet d39f042afb954daa923502ef12377bd5 (DEFAULT_TABLE table=default.Table_cccdd46e80f149d58dbedf595291b449 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.579562 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59855ab3e42b4e019b769007ccbbc360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.579842 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39f042afb954daa923502ef12377bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.581190 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 59855ab3e42b4e019b769007ccbbc360 (DEFAULT_TABLE table=default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.581568 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59855ab3e42b4e019b769007ccbbc360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.583858 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94fb9d53044d796e2973143d888fb (DEFAULT_TABLE table=default.Table_7fcef0fe6b204a57aef33be1149303db [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.584205 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94fb9d53044d796e2973143d888fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.584672 5114 ts_tablet_manager.cc:1875] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.585247 4778 tablet_service.cc:1467] Processing CreateTablet for tablet b28f66fe53064a46a0b98b78aa77115c (DEFAULT_TABLE table=default.Table_2f1e52a8b5d44129acbcd3c03132861e [id=2ad7c9ff8c2d4cb19b38ebf6008349d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.585629 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28f66fe53064a46a0b98b78aa77115c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.587713 5115 ts_tablet_manager.cc:1909] T b487131ac3a842009b3fe54578ea091d P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.588200 5115 tablet_replica.cc:330] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.588404 5115 raft_consensus.cc:2229] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.589231 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 7178760c21904207b6c8317093871f63 (DEFAULT_TABLE table=default.Table_5a2971f0db2147bb875eba0e1a871f95 [id=07e16e4056424605a70ccfaa25a24613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.589406 5115 raft_consensus.cc:2258] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.589643 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178760c21904207b6c8317093871f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.590246 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6681572bb5e5469bb236a2eb7975fd0a" } modify_external_catalogs: true
I20240503 02:12:21.590438 5106 tablet_bootstrap.cc:654] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.590452 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6681572bb5e5469bb236a2eb7975fd0a" } modify_external_catalogs: true
I20240503 02:12:21.591802 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b487131ac3a842009b3fe54578ea091d (table default.Table_f1fb4997ae544428a3067a3b9ae447dc [id=a24019c2303b40a2adab179b7ea5ca1b]) successfully deleted
I20240503 02:12:21.592372 4272 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:21.593243 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 95681e67db1e4485876d530133e8a514 (DEFAULT_TABLE table=default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.593640 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95681e67db1e4485876d530133e8a514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.594102 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:21.597119 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 48e713ee37d042309166b0f118e14986 (DEFAULT_TABLE table=default.Table_37e1baa613ca4b10bb2727227e93163b [id=b4c6a45a47e647b1947d0d7f2ea30afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.597517 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e713ee37d042309166b0f118e14986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.598377 5277 raft_consensus.cc:482] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.598532 5277 raft_consensus.cc:504] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.599539 5277 leader_election.cc:288] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:21.600077 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:21.600338 4529 raft_consensus.cc:2454] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:21.600880 4746 leader_election.cc:302] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:21.601192 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94fb9d53044d796e2973143d888fb (DEFAULT_TABLE table=default.Table_7fcef0fe6b204a57aef33be1149303db [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.601379 5103 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.601307 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:21.601542 5103 ts_tablet_manager.cc:1367] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.000s
I20240503 02:12:21.601572 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94fb9d53044d796e2973143d888fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:21.601861 4746 leader_election.cc:341] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.602177 5277 raft_consensus.cc:2781] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:21.602291 5277 raft_consensus.cc:482] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.602371 5277 raft_consensus.cc:3037] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.602326 5103 raft_consensus.cc:348] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.602540 5103 raft_consensus.cc:374] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.602612 5103 raft_consensus.cc:729] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.604818 5106 tablet_bootstrap.cc:492] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.605319 5106 ts_tablet_manager.cc:1367] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20240503 02:12:21.607539 5159 tablet_replica.cc:330] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:21.607721 5159 raft_consensus.cc:2229] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.607882 5159 raft_consensus.cc:2258] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.609539 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_19707a170e734efab69018ce4a0d61f7" } modify_external_catalogs: true
I20240503 02:12:21.609768 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_19707a170e734efab69018ce4a0d61f7" } modify_external_catalogs: true
I20240503 02:12:21.610850 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fe5117d4d59b4f0b87c32e8b27b085d9 (table default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]) successfully deleted
W20240503 02:12:21.611172 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 885129a4efc5475b97b7ac68e1db5e1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885129a4efc5475b97b7ac68e1db5e1d already in progress: creating tablet
W20240503 02:12:21.613276 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7394c9d374404fb199ab6ff170d7febe consensus state has no leader
I20240503 02:12:21.613900 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_247b02731e084bc5985d5f3d2dbac760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.614370 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c1c8837504ad4efd959e72fc2f52c010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.614513 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_247b02731e084bc5985d5f3d2dbac760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.614797 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1c8837504ad4efd959e72fc2f52c010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.616958 5103 consensus_queue.cc:260] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.621011 5106 raft_consensus.cc:348] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.626183 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_174379bd4f48483fa1baddfe455090de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.627012 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:21.627358 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_174379bd4f48483fa1baddfe455090de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.628690 5103 ts_tablet_manager.cc:1398] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:12:21.626785 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7fcef0fe6b204a57aef33be1149303db" } new_table_name: "default.Table_1aea91e895ce4610be1ebf9ee910958b" modify_external_catalogs: true
I20240503 02:12:21.631814 5106 raft_consensus.cc:374] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.632107 5115 ts_tablet_manager.cc:1875] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.633215 5106 raft_consensus.cc:729] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.624498 5159 ts_tablet_manager.cc:1875] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.634696 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_29408dd96a06477d99f95b091aa7f728" } modify_external_catalogs: true
I20240503 02:12:21.635144 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_29408dd96a06477d99f95b091aa7f728" } modify_external_catalogs: true
I20240503 02:12:21.635288 5106 consensus_queue.cc:260] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.636744 5114 ts_tablet_manager.cc:1888] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.636955 5114 log.cc:1196] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b7daa5af6c8044178b9f84aaba4cca33
I20240503 02:12:21.637341 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 59855ab3e42b4e019b769007ccbbc360 (DEFAULT_TABLE table=default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.638403 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1ea22186c7c9467c8a58b89fdf817b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.638676 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59855ab3e42b4e019b769007ccbbc360. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:21.639003 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ea22186c7c9467c8a58b89fdf817b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.641023 5106 ts_tablet_manager.cc:1398] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20240503 02:12:21.641126 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.641579 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet a295609170a8482f8373dbf817b1eea1 with delete_type TABLET_DATA_DELETED (replaced by tablet 22776c81d1a24d368abb68274cf32a0b) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.642014 5277 raft_consensus.cc:504] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.643128 5277 leader_election.cc:288] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:21.643707 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:21.643919 4529 raft_consensus.cc:3037] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.644440 5106 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.646663 4529 raft_consensus.cc:2454] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:21.647083 5103 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.647248 4746 leader_election.cc:302] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:21.647668 5277 raft_consensus.cc:2781] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.648753 5103 tablet_bootstrap.cc:654] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.649493 5277 raft_consensus.cc:686] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:21.649863 5277 consensus_queue.cc:237] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.651098 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:21.652144 4746 leader_election.cc:341] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.652578 5106 tablet_bootstrap.cc:654] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.657130 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
I20240503 02:12:21.659659 5115 ts_tablet_manager.cc:1888] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.659804 5115 log.cc:1196] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/fc887bc4db1c4e268c4c9f23096d704c
I20240503 02:12:21.660499 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a258c9bad19a4ce393deddba14dfcdbc" } new_table_name: "default.Table_d54df163e4634a779d3bcfddb083db60" modify_external_catalogs: true
I20240503 02:12:21.661657 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.662616 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.663833 5115 ts_tablet_manager.cc:1909] T fc887bc4db1c4e268c4c9f23096d704c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.664733 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet fc887bc4db1c4e268c4c9f23096d704c (table default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]) successfully deleted
I20240503 02:12:21.665397 5115 tablet_replica.cc:330] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.665607 5115 raft_consensus.cc:2229] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.665773 5115 raft_consensus.cc:2258] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.667891 5115 ts_tablet_manager.cc:1875] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.668367 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:21.669184 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.669478 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.669613 5105 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.669678 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.669775 5105 ts_tablet_manager.cc:1367] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.249s user 0.003s sys 0.006s
I20240503 02:12:21.670514 5105 raft_consensus.cc:348] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.671514 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6f061523dfc54120ae18f90ddd6c06f4" } new_table_name: "default.Table_92359e32d7a04637bfb894a7f9e1fb42" modify_external_catalogs: true
I20240503 02:12:21.672040 5114 ts_tablet_manager.cc:1909] T b7daa5af6c8044178b9f84aaba4cca33 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.673084 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b7daa5af6c8044178b9f84aaba4cca33 (table default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]) successfully deleted
I20240503 02:12:21.677336 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e625e02b764ff2b7279b35dd10b08c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.677803 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.678285 5286 raft_consensus.cc:482] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.678452 5105 raft_consensus.cc:374] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.678558 5105 raft_consensus.cc:729] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.678494 5286 raft_consensus.cc:504] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.678817 5105 consensus_queue.cc:260] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.679256 5114 tablet_replica.cc:330] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.679513 5114 raft_consensus.cc:2229] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.679720 5114 raft_consensus.cc:2258] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.679275 5286 leader_election.cc:288] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:21.679829 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced251fecca84866ab3ff15bd225e2b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:21.680321 4462 leader_election.cc:341] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:21.680275 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced251fecca84866ab3ff15bd225e2b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:21.680490 4813 raft_consensus.cc:2454] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:21.681608 4462 leader_election.cc:302] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:21.681814 4462 raft_consensus.cc:2670] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:21.682515 5115 ts_tablet_manager.cc:1888] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.682679 5115 log.cc:1196] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/bc0e75639a4f4f168f9d833b090ae76a
I20240503 02:12:21.683053 5115 ts_tablet_manager.cc:1909] T bc0e75639a4f4f168f9d833b090ae76a P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.683382 5159 ts_tablet_manager.cc:1888] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.683539 5159 log.cc:1196] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:21.683709 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet bc0e75639a4f4f168f9d833b090ae76a (table default.Table_5c5ea1ab22594d6b81fbd5b536231007 [id=9947c6f15d2444ca895eea2f96df51ae]) successfully deleted
I20240503 02:12:21.683907 5159 ts_tablet_manager.cc:1909] T 14fd4fb88436498999b5f339052ac0b6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.685084 5105 ts_tablet_manager.cc:1398] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 02:12:21.685441 5105 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.691854 5105 tablet_bootstrap.cc:654] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.693686 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 14fd4fb88436498999b5f339052ac0b6 (table default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]) successfully deleted
W20240503 02:12:21.694111 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2f053537029a44d5a519b612d09d19a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f053537029a44d5a519b612d09d19a1 already in progress: creating tablet
I20240503 02:12:21.694680 5115 tablet_replica.cc:330] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.694882 5115 raft_consensus.cc:2229] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.695072 5115 raft_consensus.cc:2258] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:21.695719 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a already in progress: creating tablet
W20240503 02:12:21.695969 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3bc9fe2302024e5baa74d32941d70d29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bc9fe2302024e5baa74d32941d70d29 already in progress: creating tablet
I20240503 02:12:21.697592 5115 ts_tablet_manager.cc:1875] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.699935 5105 tablet_bootstrap.cc:492] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.700146 5105 ts_tablet_manager.cc:1367] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20240503 02:12:21.700182 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a74f4816cad04a9eb5934aaba86c2fbc
I20240503 02:12:21.700942 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.701195 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.701527 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:21.701601 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a74f4816cad04a9eb5934aaba86c2fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74f4816cad04a9eb5934aaba86c2fbc already in progress: creating tablet
I20240503 02:12:21.701674 4275 catalog_manager.cc:5484] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.702071 5105 raft_consensus.cc:348] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.702176 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:21.702286 5105 raft_consensus.cc:374] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.702378 5105 raft_consensus.cc:729] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.702661 5105 consensus_queue.cc:260] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.704838 5105 ts_tablet_manager.cc:1398] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:21.705299 5105 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.706296 4269 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 9b0893f9a64143739ddef9495cc20ce4
I20240503 02:12:21.706725 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4cc509b619474ecd8fc3f145f028ce24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.707320 5105 tablet_bootstrap.cc:654] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:21.707345 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cc509b619474ecd8fc3f145f028ce24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.709563 5105 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.709792 5105 ts_tablet_manager.cc:1367] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:12:21.711126 5115 ts_tablet_manager.cc:1888] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.711253 5115 log.cc:1196] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b7daa5af6c8044178b9f84aaba4cca33
I20240503 02:12:21.711912 5105 raft_consensus.cc:348] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.712205 5105 raft_consensus.cc:374] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.712340 5105 raft_consensus.cc:729] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.712680 5105 consensus_queue.cc:260] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.714735 5105 ts_tablet_manager.cc:1398] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:21.715268 5105 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.717594 5115 ts_tablet_manager.cc:1909] T b7daa5af6c8044178b9f84aaba4cca33 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.718545 5105 tablet_bootstrap.cc:654] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.720220 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b7daa5af6c8044178b9f84aaba4cca33 (table default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]) successfully deleted
I20240503 02:12:21.720870 5114 ts_tablet_manager.cc:1875] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.721679 5115 tablet_replica.cc:330] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.721985 5115 raft_consensus.cc:2229] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.722199 5115 raft_consensus.cc:2258] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.723330 5115 ts_tablet_manager.cc:1875] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.726871 5105 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.727097 5105 ts_tablet_manager.cc:1367] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 02:12:21.728250 5105 raft_consensus.cc:348] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.728534 5105 raft_consensus.cc:374] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.728673 5105 raft_consensus.cc:729] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.729013 5105 consensus_queue.cc:260] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.730875 5105 ts_tablet_manager.cc:1398] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:21.730741 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_36b1bbc5ec49474881c8c6eca0df22ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.731303 5105 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:21.731400 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b1bbc5ec49474881c8c6eca0df22ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.733438 5105 tablet_bootstrap.cc:654] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.738087 5115 ts_tablet_manager.cc:1888] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.738371 5115 log.cc:1196] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/a295609170a8482f8373dbf817b1eea1
I20240503 02:12:21.738914 5115 ts_tablet_manager.cc:1909] T a295609170a8482f8373dbf817b1eea1 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.740303 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet a295609170a8482f8373dbf817b1eea1 (table default.Table_98ecda629a0e48c3bcb69d91efbfe2f4 [id=07ea56ce8d3749dfbcbc1a79ea84693c]) successfully deleted
W20240503 02:12:21.741232 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3bc9fe2302024e5baa74d32941d70d29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bc9fe2302024e5baa74d32941d70d29 already in progress: creating tablet
I20240503 02:12:21.741660 5115 tablet_replica.cc:330] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.741683 5106 tablet_bootstrap.cc:492] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.741909 5106 ts_tablet_manager.cc:1367] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.100s user 0.015s sys 0.028s
I20240503 02:12:21.742026 5115 raft_consensus.cc:2229] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.742262 5115 raft_consensus.cc:2258] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.742754 5106 raft_consensus.cc:348] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.742888 5105 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.742969 5106 raft_consensus.cc:374] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.743062 5105 ts_tablet_manager.cc:1367] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 02:12:21.743072 5106 raft_consensus.cc:729] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.743285 5115 ts_tablet_manager.cc:1875] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.743309 5106 consensus_queue.cc:260] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.745077 5106 ts_tablet_manager.cc:1398] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:21.745030 5105 raft_consensus.cc:348] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.745401 5106 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.745533 5105 raft_consensus.cc:374] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.745669 5105 raft_consensus.cc:729] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.746104 5105 consensus_queue.cc:260] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.748570 5105 ts_tablet_manager.cc:1398] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:21.749122 5105 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:21.750730 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3db94fb9d53044d796e2973143d888fb consensus state has no leader
I20240503 02:12:21.751147 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_05e6a7077cf54c3ebcfb08151b563cdc" } new_table_name: "default.Table_04781eea28204bf196d694bd2986e194" modify_external_catalogs: true
I20240503 02:12:21.752213 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 29e5c40f1f6b40308eed26b64d047494 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.752655 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.755005 5105 tablet_bootstrap.cc:654] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.758579 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 24737f8a8ca74996b9f54d49d12b9288 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.758761 5103 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.759092 5103 ts_tablet_manager.cc:1367] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.000s
I20240503 02:12:21.760268 5103 raft_consensus.cc:348] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.760571 5103 raft_consensus.cc:374] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.760675 5103 raft_consensus.cc:729] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.760990 5103 consensus_queue.cc:260] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.765801 5105 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.766024 5105 ts_tablet_manager.cc:1367] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20240503 02:12:21.766139 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5117d4d59b4f0b87c32e8b27b085d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.766539 5106 tablet_bootstrap.cc:654] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.767027 5105 raft_consensus.cc:348] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.767326 5105 raft_consensus.cc:374] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.767478 5105 raft_consensus.cc:729] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.767782 5115 ts_tablet_manager.cc:1888] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.767923 5115 log.cc:1196] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/21e625e02b764ff2b7279b35dd10b08c
I20240503 02:12:21.767849 5105 consensus_queue.cc:260] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.768304 5115 ts_tablet_manager.cc:1909] T 21e625e02b764ff2b7279b35dd10b08c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.768875 5103 ts_tablet_manager.cc:1398] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 02:12:21.769361 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 21e625e02b764ff2b7279b35dd10b08c (table default.Table_7b1ece612f9744a79e3c49f09e7ddb24 [id=a8708feb75e1400bbfb86192505008ce]) successfully deleted
W20240503 02:12:21.770239 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a already in progress: creating tablet
W20240503 02:12:21.770967 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a74f4816cad04a9eb5934aaba86c2fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74f4816cad04a9eb5934aaba86c2fbc already in progress: creating tablet
I20240503 02:12:21.771252 5115 tablet_replica.cc:330] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:21.771466 5115 raft_consensus.cc:2229] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.771477 5105 ts_tablet_manager.cc:1398] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:21.771616 5115 raft_consensus.cc:2258] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.772804 5105 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.773348 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:21.774374 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.774523 5105 tablet_bootstrap.cc:654] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.774631 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.775149 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:21.776073 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ea7afb429d64497abcd9f1d0910651c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7afb429d64497abcd9f1d0910651c3 already in progress: creating tablet
I20240503 02:12:21.785048 5115 ts_tablet_manager.cc:1875] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.788480 5103 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.794963 5105 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.795229 5105 ts_tablet_manager.cc:1367] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.013s
I20240503 02:12:21.796020 5105 raft_consensus.cc:348] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.796301 5105 raft_consensus.cc:374] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.796406 5105 raft_consensus.cc:729] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.802647 5115 ts_tablet_manager.cc:1888] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:21.804899 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d271d2c112ba44b68d2933a6ebfe7a18 consensus state has no leader
I20240503 02:12:21.805491 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_acbebd20514e4b8291c208833d606eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.806140 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acbebd20514e4b8291c208833d606eb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.805518 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_36b7e35193f8454d896de4043269a0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.806905 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b7e35193f8454d896de4043269a0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.807534 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91798b4bc2f74b9289425531937a78ce has no consensus state
I20240503 02:12:21.809068 5105 consensus_queue.cc:260] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.809861 5105 ts_tablet_manager.cc:1398] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 02:12:21.813277 5105 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.816617 5105 tablet_bootstrap.cc:654] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.818387 5103 tablet_bootstrap.cc:654] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.819692 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_92a91f0f1d6445e7bfb5cd31751fd4e5" } new_table_name: "default.Table_18dc31f244e94d6ab3c576eaae5e408b" modify_external_catalogs: true
I20240503 02:12:21.821058 5114 ts_tablet_manager.cc:1888] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.823066 5114 log.cc:1196] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:21.823480 5114 ts_tablet_manager.cc:1909] T ced251fecca84866ab3ff15bd225e2b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.824604 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet ced251fecca84866ab3ff15bd225e2b7 (table default.Table_4227e90304ee491ca298dabd17aa7b44 [id=536a3d22f6e34248b7936ba2e7a05fd5]) successfully deleted
I20240503 02:12:21.824832 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_919992b1410a4a6494af1ad6995d7b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.825403 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_919992b1410a4a6494af1ad6995d7b93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.824524 5114 tablet_replica.cc:330] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.826946 5114 raft_consensus.cc:2229] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.827149 5114 raft_consensus.cc:2258] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.835978 5114 ts_tablet_manager.cc:1875] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.836313 5115 log.cc:1196] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/fe5117d4d59b4f0b87c32e8b27b085d9
W20240503 02:12:21.841732 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8750b02c5936412db525d2d3e77befeb consensus state has no leader
I20240503 02:12:21.842039 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet ea7afb429d64497abcd9f1d0910651c3 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:21.843652 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet d022b5df4a914e439fff918f70c34157 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:21.845726 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f146f7cce0b24d3087bfe0973e2b7ef0 consensus state has no leader
I20240503 02:12:21.837756 5103 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.846957 5105 tablet_bootstrap.cc:492] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.846961 5103 ts_tablet_manager.cc:1367] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20240503 02:12:21.847119 5105 ts_tablet_manager.cc:1367] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.003s
I20240503 02:12:21.847857 5103 raft_consensus.cc:348] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.847860 5105 raft_consensus.cc:348] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.848078 5105 raft_consensus.cc:374] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.848098 5103 raft_consensus.cc:374] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.848168 5105 raft_consensus.cc:729] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.848168 5103 raft_consensus.cc:729] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.848451 5105 consensus_queue.cc:260] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.848460 5103 consensus_queue.cc:260] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:21.849037 5103 ts_tablet_manager.cc:1398] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.849421 5103 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.851505 5105 ts_tablet_manager.cc:1398] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:21.855491 5103 tablet_bootstrap.cc:654] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.858211 5103 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:21.858381 5103 ts_tablet_manager.cc:1367] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:21.859184 5103 raft_consensus.cc:348] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.859406 5103 raft_consensus.cc:374] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.859519 5103 raft_consensus.cc:729] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.859838 5103 consensus_queue.cc:260] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.860469 5103 ts_tablet_manager.cc:1398] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.860846 5103 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:21.861315 5105 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.862984 5103 tablet_bootstrap.cc:654] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.863045 5105 tablet_bootstrap.cc:654] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.836899 5115 ts_tablet_manager.cc:1909] T fe5117d4d59b4f0b87c32e8b27b085d9 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:21.843087 5106 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:21.864207 5106 ts_tablet_manager.cc:1367] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.119s user 0.018s sys 0.003s
I20240503 02:12:21.865084 5106 raft_consensus.cc:348] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.865325 5106 raft_consensus.cc:374] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.865393 5106 raft_consensus.cc:729] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.866267 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d510e8f3fde248f29d85de95822d83af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:21.866775 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet fe5117d4d59b4f0b87c32e8b27b085d9 (table default.Table_b741361e02de4b5aaed514fe7cd99f58 [id=dc859fc521b24a629a60d53618651f4c]) successfully deleted
W20240503 02:12:21.866930 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d510e8f3fde248f29d85de95822d83af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:21.867540 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ea7afb429d64497abcd9f1d0910651c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7afb429d64497abcd9f1d0910651c3 already in progress: creating tablet
I20240503 02:12:21.870594 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_df633d54f27d4e71b052cbf0582d8ba2" } modify_external_catalogs: true
I20240503 02:12:21.870810 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_df633d54f27d4e71b052cbf0582d8ba2" } modify_external_catalogs: true
I20240503 02:12:21.871363 5114 ts_tablet_manager.cc:1888] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.871508 5114 log.cc:1196] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:21.871922 5114 ts_tablet_manager.cc:1909] T 003c9c3e0a1d400c94bfaefada2e9faa P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:21.873091 5114 tablet_replica.cc:330] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:21.873312 5114 raft_consensus.cc:2229] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.873493 5114 raft_consensus.cc:2258] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.873922 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 003c9c3e0a1d400c94bfaefada2e9faa (table default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]) successfully deleted
I20240503 02:12:21.874441 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2193a952b28f4000b0472d610e708f83" } modify_external_catalogs: true
I20240503 02:12:21.874640 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2193a952b28f4000b0472d610e708f83" } modify_external_catalogs: true
I20240503 02:12:21.877025 5105 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.877218 5105 ts_tablet_manager.cc:1367] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240503 02:12:21.877627 4787 tablet_service.cc:1467] Processing CreateTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 (DEFAULT_TABLE table=default.Table_c4e56102ddd84ad58342dbdfb695e622 [id=e22df2e2673c45cbb1b32201a728333c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.878048 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9511e4de6c942b9b9e0790c88d3d543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.878129 5105 raft_consensus.cc:348] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.878360 5105 raft_consensus.cc:374] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.878435 5105 raft_consensus.cc:729] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.878690 5105 consensus_queue.cc:260] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.880687 5105 ts_tablet_manager.cc:1398] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:21.881031 5105 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.881234 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 (DEFAULT_TABLE table=default.Table_3c3549c20fa149b78d707a1a111895d7 [id=5cbfe61d82174908b3b00ff5d287eb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.881616 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4225c1bc133d4e18a4a47fd58820ca10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.882736 5105 tablet_bootstrap.cc:654] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:21.889449 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a already in progress: creating tablet
I20240503 02:12:21.891774 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b7daa5af6c8044178b9f84aaba4cca33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.891914 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 (DEFAULT_TABLE table=default.Table_308844f35cb448378721f1da35d38c48 [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.891999 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.892092 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 87713e899d2148308316d807a6d76921 (DEFAULT_TABLE table=default.Table_0fbcd3f402c1447f86400a7052a931a3 [id=31669972ad634736b38958103d7b6c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.892249 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.892392 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9090b11c8a4bf0b4b92523b9431b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.892481 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.892630 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87713e899d2148308316d807a6d76921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.892679 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet a295609170a8482f8373dbf817b1eea1 with delete_type TABLET_DATA_DELETED (replaced by tablet 22776c81d1a24d368abb68274cf32a0b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.893029 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.893471 4780 tablet_service.cc:1467] Processing CreateTablet for tablet ea5e4db5204e4acf8719100d02d533fb (DEFAULT_TABLE table=default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.893867 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5e4db5204e4acf8719100d02d533fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.896791 5159 tablet_replica.cc:330] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:21.897074 5159 raft_consensus.cc:2229] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.897256 5159 raft_consensus.cc:2258] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.891918 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.892874 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.902197 5159 ts_tablet_manager.cc:1875] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.906150 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 556f8938f35f42bfbbfb27f829653d45 (DEFAULT_TABLE table=default.Table_72645b71fd5b41039586175c6982ccec [id=2bc553d540a14a91ab3d452e8edaa90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.906812 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f8938f35f42bfbbfb27f829653d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.908402 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 (DEFAULT_TABLE table=default.Table_143457686a2e414c8d11b68e09860a08 [id=41995c76abc744c1b75efd4e1a4b4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.909379 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a90ebcf29004d1a925b69fa20998dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.910405 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f556497cf91b4b6cb71095aceb0c3a58" } modify_external_catalogs: true
I20240503 02:12:21.910697 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f556497cf91b4b6cb71095aceb0c3a58" } modify_external_catalogs: true
I20240503 02:12:21.912686 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_98ecda629a0e48c3bcb69d91efbfe2f4" } modify_external_catalogs: true
I20240503 02:12:21.913004 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_98ecda629a0e48c3bcb69d91efbfe2f4" } modify_external_catalogs: true
I20240503 02:12:21.921038 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:21.921552 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 deleted and replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:21.922116 5114 ts_tablet_manager.cc:1875] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:21.930353 5105 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:21.930541 5105 ts_tablet_manager.cc:1367] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.021s
I20240503 02:12:21.931638 5105 raft_consensus.cc:348] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.931929 5105 raft_consensus.cc:374] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.932160 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.932039 5105 raft_consensus.cc:729] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.932631 5105 consensus_queue.cc:260] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.932744 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_27e6472c5d7746f59ee6399fa6db4d3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.933370 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27e6472c5d7746f59ee6399fa6db4d3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.935578 5105 ts_tablet_manager.cc:1398] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:21.936094 5159 ts_tablet_manager.cc:1888] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:21.936324 5159 log.cc:1196] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b7daa5af6c8044178b9f84aaba4cca33
I20240503 02:12:21.936713 5159 ts_tablet_manager.cc:1909] T b7daa5af6c8044178b9f84aaba4cca33 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:21.936944 5105 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:21.938686 5105 tablet_bootstrap.cc:654] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.942035 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.876457 4647 tablet_service.cc:1467] Processing CreateTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 (DEFAULT_TABLE table=default.Table_c4e56102ddd84ad58342dbdfb695e622 [id=e22df2e2673c45cbb1b32201a728333c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.942513 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9511e4de6c942b9b9e0790c88d3d543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.943301 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 95681e67db1e4485876d530133e8a514
I20240503 02:12:21.886873 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 (DEFAULT_TABLE table=default.Table_3c3549c20fa149b78d707a1a111895d7 [id=5cbfe61d82174908b3b00ff5d287eb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.944451 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4225c1bc133d4e18a4a47fd58820ca10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.886924 4637 tablet_service.cc:1467] Processing CreateTablet for tablet ea5e4db5204e4acf8719100d02d533fb (DEFAULT_TABLE table=default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.949352 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5e4db5204e4acf8719100d02d533fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.950215 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:21.865929 5106 consensus_queue.cc:260] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:21.953059 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.901335 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 87713e899d2148308316d807a6d76921 (DEFAULT_TABLE table=default.Table_0fbcd3f402c1447f86400a7052a931a3 [id=31669972ad634736b38958103d7b6c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.953822 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87713e899d2148308316d807a6d76921. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:21.953795 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 95681e67db1e4485876d530133e8a514 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95681e67db1e4485876d530133e8a514 already in progress: creating tablet
I20240503 02:12:21.955473 5106 ts_tablet_manager.cc:1398] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20240503 02:12:21.957369 5106 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:21.901477 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 (DEFAULT_TABLE table=default.Table_308844f35cb448378721f1da35d38c48 [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.964295 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9090b11c8a4bf0b4b92523b9431b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.964363 4494 tablet_service.cc:1467] Processing CreateTablet for tablet ea5e4db5204e4acf8719100d02d533fb (DEFAULT_TABLE table=default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.966498 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5e4db5204e4acf8719100d02d533fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.966719 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 (DEFAULT_TABLE table=default.Table_143457686a2e414c8d11b68e09860a08 [id=41995c76abc744c1b75efd4e1a4b4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.967077 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a90ebcf29004d1a925b69fa20998dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.968030 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 556f8938f35f42bfbbfb27f829653d45 (DEFAULT_TABLE table=default.Table_72645b71fd5b41039586175c6982ccec [id=2bc553d540a14a91ab3d452e8edaa90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.968456 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f8938f35f42bfbbfb27f829653d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.905530 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 556f8938f35f42bfbbfb27f829653d45 (DEFAULT_TABLE table=default.Table_72645b71fd5b41039586175c6982ccec [id=2bc553d540a14a91ab3d452e8edaa90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.973423 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f8938f35f42bfbbfb27f829653d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.976106 5106 tablet_bootstrap.cc:654] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.905764 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 (DEFAULT_TABLE table=default.Table_143457686a2e414c8d11b68e09860a08 [id=41995c76abc744c1b75efd4e1a4b4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.931201 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:21.977406 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a90ebcf29004d1a925b69fa20998dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.983503 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 24737f8a8ca74996b9f54d49d12b9288
I20240503 02:12:21.984212 4270 catalog_manager.cc:6554] ReplaceTablet: tablet 24737f8a8ca74996b9f54d49d12b9288 deleted and replaced by tablet c2efdbfa909c475d95e99cf24206608f
I20240503 02:12:21.985198 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d022b5df4a914e439fff918f70c34157
I20240503 02:12:21.985671 4273 catalog_manager.cc:6554] ReplaceTablet: tablet d022b5df4a914e439fff918f70c34157 deleted and replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd
W20240503 02:12:21.985760 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 already in progress: creating tablet
I20240503 02:12:21.986075 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b7daa5af6c8044178b9f84aaba4cca33 (table default.Table_bbe8e0e2f256445e9a2e5549f6a0fb6b [id=553aad414c46421cb2068fd9c034cb2e]) successfully deleted
I20240503 02:12:21.986434 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 (DEFAULT_TABLE table=default.Table_308844f35cb448378721f1da35d38c48 [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.987342 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 (DEFAULT_TABLE table=default.Table_3c3549c20fa149b78d707a1a111895d7 [id=5cbfe61d82174908b3b00ff5d287eb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.987946 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 87713e899d2148308316d807a6d76921 (DEFAULT_TABLE table=default.Table_0fbcd3f402c1447f86400a7052a931a3 [id=31669972ad634736b38958103d7b6c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.988122 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_be9606643d6b471db6114211488d37e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.988754 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be9606643d6b471db6114211488d37e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.989941 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.990512 4504 tablet_service.cc:1467] Processing CreateTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 (DEFAULT_TABLE table=default.Table_c4e56102ddd84ad58342dbdfb695e622 [id=e22df2e2673c45cbb1b32201a728333c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.990792 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4225c1bc133d4e18a4a47fd58820ca10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.991062 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0fbcd3f402c1447f86400a7052a931a3" } new_table_name: "default.Table_4702ebe6617c4aebb1264298076c6a4d" modify_external_catalogs: true
I20240503 02:12:21.992389 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8b04b325acf94deea5af5e4a9660f7d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:21.992903 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b04b325acf94deea5af5e4a9660f7d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:21.995674 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9090b11c8a4bf0b4b92523b9431b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.997165 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:21.997349 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.002398 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9511e4de6c942b9b9e0790c88d3d543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.006677 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87713e899d2148308316d807a6d76921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.008267 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b91abaf51abb4c2ca65412c3771c1ef2
I20240503 02:12:22.009379 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 959d998ec4ec4a52a6031650f2da4691
I20240503 02:12:22.010198 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.010936 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.013607 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_31852ddf01b44f4a88e80b4216b590c9" } new_table_name: "default.Table_7566b876f57547599a98c4f55fbfc427" modify_external_catalogs: true
I20240503 02:12:22.015870 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1ea22186c7c9467c8a58b89fdf817b19" } modify_external_catalogs: true
I20240503 02:12:22.016119 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1ea22186c7c9467c8a58b89fdf817b19" } modify_external_catalogs: true
I20240503 02:12:22.016830 5286 raft_consensus.cc:482] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.017040 5286 raft_consensus.cc:504] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.018831 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57f3833a9240c88e0de1b486fae265" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.019158 4813 raft_consensus.cc:2454] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:22.019917 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57f3833a9240c88e0de1b486fae265" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.020726 5256 raft_consensus.cc:482] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.020956 5256 raft_consensus.cc:504] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.021814 5256 leader_election.cc:288] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.022995 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:22.023645 4608 leader_election.cc:341] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.024055 5286 leader_election.cc:288] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.026881 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 003c9c3e0a1d400c94bfaefada2e9faa from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.028077 4462 leader_election.cc:302] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:22.028435 4462 leader_election.cc:341] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.032516 4671 raft_consensus.cc:3037] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.037361 5277 consensus_queue.cc:1035] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.047999 5286 raft_consensus.cc:2781] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.048410 5286 raft_consensus.cc:482] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.048543 5286 raft_consensus.cc:3037] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.048856 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_34fcaf75826241bea9fc24cb331f823e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.049576 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34fcaf75826241bea9fc24cb331f823e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.048931 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.052819 4529 raft_consensus.cc:2454] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:22.054479 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b6ddecc9e022479595b1ce5c50323184" } new_table_name: "default.Table_fd0e6dfdbbc14c7599538eb05a30083c" modify_external_catalogs: true
I20240503 02:12:22.055644 5286 raft_consensus.cc:504] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.056372 5286 leader_election.cc:288] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:22.058130 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87713e899d2148308316d807a6d76921 consensus state has no leader
I20240503 02:12:22.058667 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_af6ff741f769452798eade9274479e3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.060879 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af6ff741f769452798eade9274479e3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.060351 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0414bb3004e94f78a6ff888c2ef0cb8b" } new_table_name: "default.Table_e0cba6fcfb4c468dbf573bd12deb9897" modify_external_catalogs: true
I20240503 02:12:22.059753 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.063284 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.064246 4608 leader_election.cc:302] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:22.065069 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57f3833a9240c88e0de1b486fae265" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:22.065337 4813 raft_consensus.cc:3037] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.068161 4813 raft_consensus.cc:2454] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:22.068807 5256 raft_consensus.cc:2781] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.068949 5256 raft_consensus.cc:482] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.069056 5256 raft_consensus.cc:3037] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.071738 5256 raft_consensus.cc:504] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.073007 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57f3833a9240c88e0de1b486fae265" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:22.073751 4462 leader_election.cc:341] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.073989 4462 leader_election.cc:302] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:22.074892 5256 leader_election.cc:288] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.075448 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:22.075665 4529 raft_consensus.cc:3037] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.078440 4529 raft_consensus.cc:2454] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:22.079066 4608 leader_election.cc:302] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.080008 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:22.080543 5286 raft_consensus.cc:2781] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:22.081094 4608 leader_election.cc:341] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.081634 5256 raft_consensus.cc:2781] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.081729 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9b099c5cc2d14c2ba0fad6df023e947c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.082363 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b099c5cc2d14c2ba0fad6df023e947c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:22.083796 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78b21a3010a4470288fbdd684ef64e77 consensus state has no leader
I20240503 02:12:22.084749 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cccdd46e80f149d58dbedf595291b449" } new_table_name: "default.Table_2c388fbe8fd148dc93f0d292381c0363" modify_external_catalogs: true
I20240503 02:12:22.086663 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b94a5914ab6549d1a420331f4fa47659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.087179 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b94a5914ab6549d1a420331f4fa47659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.087781 5114 ts_tablet_manager.cc:1888] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.087988 5114 log.cc:1196] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:22.088495 5114 ts_tablet_manager.cc:1909] T 3bc9fe2302024e5baa74d32941d70d29 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:22.100142 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b080fc5ffc564420a0b7f2b7460ebbbb consensus state has no leader
I20240503 02:12:22.100698 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 22776c81d1a24d368abb68274cf32a0b
W20240503 02:12:22.101914 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d39f042afb954daa923502ef12377bd5 consensus state has no leader
W20240503 02:12:22.106192 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41d44fe8f6334572996665bdec60e2e5 consensus state has no leader
I20240503 02:12:22.113765 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.116020 5277 raft_consensus.cc:482] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.116215 5277 raft_consensus.cc:504] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.116997 5277 leader_election.cc:288] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.117970 5105 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.118151 5105 ts_tablet_manager.cc:1367] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.181s user 0.051s sys 0.043s
I20240503 02:12:22.118227 5159 tablet_replica.cc:330] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:22.118464 5159 raft_consensus.cc:2229] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.118669 5159 raft_consensus.cc:2258] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.119261 5105 raft_consensus.cc:348] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.119632 5105 raft_consensus.cc:374] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.119724 5106 tablet_bootstrap.cc:492] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.119774 5105 raft_consensus.cc:729] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.119877 5106 ts_tablet_manager.cc:1367] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.163s user 0.002s sys 0.003s
W20240503 02:12:22.120012 4746 leader_election.cc:341] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: a295609170a8482f8373dbf817b1eea1
I20240503 02:12:22.120172 5105 consensus_queue.cc:260] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.121064 5105 ts_tablet_manager.cc:1398] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:22.121451 5105 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:22.124370 4746 leader_election.cc:341] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: a295609170a8482f8373dbf817b1eea1
I20240503 02:12:22.124548 4746 leader_election.cc:302] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.125423 5277 raft_consensus.cc:2726] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.125869 5106 raft_consensus.cc:348] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.126130 5106 raft_consensus.cc:374] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.126250 5106 raft_consensus.cc:729] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.126533 5106 consensus_queue.cc:260] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.127077 5106 ts_tablet_manager.cc:1398] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:12:22.127067 5103 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.127393 5106 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.128973 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c9a280da9b284233bbd84cc8bb291cca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.130609 5103 ts_tablet_manager.cc:1367] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.270s user 0.004s sys 0.000s
W20240503 02:12:22.131392 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9a280da9b284233bbd84cc8bb291cca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.132627 5256 raft_consensus.cc:686] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:22.132819 5286 raft_consensus.cc:686] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:22.134896 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.138571 5105 tablet_bootstrap.cc:654] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.151757 5286 consensus_queue.cc:237] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.134788 5106 tablet_bootstrap.cc:654] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.153844 5256 consensus_queue.cc:237] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.155910 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.160039 5103 raft_consensus.cc:348] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.160400 5103 raft_consensus.cc:374] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.160589 5103 raft_consensus.cc:729] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.161006 5103 consensus_queue.cc:260] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.162998 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.164137 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.165520 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.166394 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.166541 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:22.167359 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.168073 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.168779 5159 ts_tablet_manager.cc:1875] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.173830 5106 tablet_bootstrap.cc:492] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.174037 5106 ts_tablet_manager.cc:1367] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.009s
I20240503 02:12:22.174935 5106 raft_consensus.cc:348] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.175194 5106 raft_consensus.cc:374] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.175093 5201 consensus_queue.cc:1035] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.175282 5106 raft_consensus.cc:729] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.175541 5106 consensus_queue.cc:260] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.176153 5106 ts_tablet_manager.cc:1398] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:22.176522 5106 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.178372 5106 tablet_bootstrap.cc:654] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.181612 5105 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.181876 5105 ts_tablet_manager.cc:1367] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.012s
I20240503 02:12:22.182811 5105 raft_consensus.cc:348] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.183125 5105 raft_consensus.cc:374] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.183290 5105 raft_consensus.cc:729] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.183298 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.183631 5105 consensus_queue.cc:260] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.184038 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.184513 5105 ts_tablet_manager.cc:1398] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:22.185076 5106 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.185497 5105 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.187271 5105 tablet_bootstrap.cc:654] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.187520 5106 ts_tablet_manager.cc:1367] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:12:22.188422 5106 raft_consensus.cc:348] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.188707 5106 raft_consensus.cc:374] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.188836 5106 raft_consensus.cc:729] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.189131 5106 consensus_queue.cc:260] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.190228 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f1cb580826c474cb8690af6403aefdf" } modify_external_catalogs: true
I20240503 02:12:22.190446 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f1cb580826c474cb8690af6403aefdf" } modify_external_catalogs: true
I20240503 02:12:22.190672 5201 raft_consensus.cc:482] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.190806 5106 ts_tablet_manager.cc:1398] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:22.190827 5201 raft_consensus.cc:504] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.191148 5106 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.191473 5201 leader_election.cc:288] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.192706 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.192854 5106 tablet_bootstrap.cc:654] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.193009 4528 raft_consensus.cc:2454] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.193594 4746 leader_election.cc:302] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.194185 5201 raft_consensus.cc:2781] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.194314 5201 raft_consensus.cc:482] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.194406 5201 raft_consensus.cc:3037] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.197527 5201 raft_consensus.cc:504] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.198684 5201 leader_election.cc:288] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.199025 5106 tablet_bootstrap.cc:492] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.199218 5106 ts_tablet_manager.cc:1367] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:12:22.200014 5106 raft_consensus.cc:348] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.200253 5106 raft_consensus.cc:374] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.200322 5106 raft_consensus.cc:729] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.200541 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:22.200544 5106 consensus_queue.cc:260] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.200773 4528 raft_consensus.cc:3037] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.201092 5106 ts_tablet_manager.cc:1398] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:22.201400 5106 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.202998 5106 tablet_bootstrap.cc:654] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.203495 4528 raft_consensus.cc:2454] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:22.204250 4746 leader_election.cc:302] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.205134 5105 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.205061 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:22.205299 4672 raft_consensus.cc:3037] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.205407 5105 ts_tablet_manager.cc:1367] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240503 02:12:22.206195 5105 raft_consensus.cc:348] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.206312 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:22.206416 5105 raft_consensus.cc:374] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.206490 5105 raft_consensus.cc:729] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.206725 5105 consensus_queue.cc:260] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.207602 5201 raft_consensus.cc:2781] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.209311 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.209517 4672 raft_consensus.cc:2454] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:22.210224 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:22.210315 5105 ts_tablet_manager.cc:1398] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:22.211376 4441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:22.211838 5105 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.213790 5105 tablet_bootstrap.cc:654] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.214272 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:22.214499 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]). Expected version 3 got 0
I20240503 02:12:22.214751 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bb18c51dac8c46b2a5ddecca6ddd34be" } modify_external_catalogs: true
I20240503 02:12:22.214982 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bb18c51dac8c46b2a5ddecca6ddd34be" } modify_external_catalogs: true
I20240503 02:12:22.216308 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_af0acc1c80404f3a875b88f459374f81" } modify_external_catalogs: true
I20240503 02:12:22.216558 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_af0acc1c80404f3a875b88f459374f81" } modify_external_catalogs: true
I20240503 02:12:22.221189 5114 tablet_replica.cc:330] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:22.222563 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.223152 5106 tablet_bootstrap.cc:492] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.223353 5106 ts_tablet_manager.cc:1367] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20240503 02:12:22.224292 5106 raft_consensus.cc:348] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.224550 5106 raft_consensus.cc:374] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.225550 5105 tablet_bootstrap.cc:492] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.225780 5105 ts_tablet_manager.cc:1367] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240503 02:12:22.225993 5103 ts_tablet_manager.cc:1398] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.095s user 0.001s sys 0.001s
I20240503 02:12:22.226925 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 53017352b8f34955b116bbfd32f04854
I20240503 02:12:22.227035 5105 raft_consensus.cc:348] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.227417 5106 raft_consensus.cc:729] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.227772 5106 consensus_queue.cc:260] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.228093 5105 raft_consensus.cc:374] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.228277 5105 raft_consensus.cc:729] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.228624 5105 consensus_queue.cc:260] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.230012 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 86d719c58974442281c7d694e1ca485e
I20240503 02:12:22.230796 5105 ts_tablet_manager.cc:1398] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:22.231169 5106 ts_tablet_manager.cc:1398] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:12:22.231578 5106 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.232699 5105 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.234566 5106 tablet_bootstrap.cc:654] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.235193 5103 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.236529 5105 tablet_bootstrap.cc:654] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.236995 5103 tablet_bootstrap.cc:654] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.244894 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.245301 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.245718 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.245965 5201 raft_consensus.cc:686] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:22.246392 5201 consensus_queue.cc:237] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.248680 5106 tablet_bootstrap.cc:492] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.248880 5106 ts_tablet_manager.cc:1367] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240503 02:12:22.249658 5106 raft_consensus.cc:348] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.249881 5106 raft_consensus.cc:374] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.249961 5106 raft_consensus.cc:729] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.250207 5106 consensus_queue.cc:260] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.250365 5114 raft_consensus.cc:2229] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:22.250762 5106 ts_tablet_manager.cc:1398] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:22.251130 5106 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.251145 5114 pending_rounds.cc:62] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:22.254179 5105 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.254510 5105 ts_tablet_manager.cc:1367] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20240503 02:12:22.255440 5105 raft_consensus.cc:348] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.255769 5105 raft_consensus.cc:374] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.255903 5105 raft_consensus.cc:729] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.256251 5105 consensus_queue.cc:260] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.256996 5105 ts_tablet_manager.cc:1398] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:22.257385 5105 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.259557 5105 tablet_bootstrap.cc:654] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.259984 5106 tablet_bootstrap.cc:654] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.260294 5114 pending_rounds.cc:68] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420793457840128 op_type: NO_OP noop_request { }
I20240503 02:12:22.260505 5114 raft_consensus.cc:2866] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:22.260618 5114 raft_consensus.cc:2258] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.263854 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 3bc9fe2302024e5baa74d32941d70d29 (table default.Table_422f8213fa2e4d269fc4499098056f64 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]) successfully deleted
W20240503 02:12:22.264181 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a74f4816cad04a9eb5934aaba86c2fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74f4816cad04a9eb5934aaba86c2fbc already in progress: creating tablet
W20240503 02:12:22.264420 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ea7afb429d64497abcd9f1d0910651c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7afb429d64497abcd9f1d0910651c3 already in progress: creating tablet
I20240503 02:12:22.264575 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_adc5b950ff6b4d9e8f02719738c475c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.265281 5114 ts_tablet_manager.cc:1875] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.266353 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_89d9f0f67e8841b69f9567d09f7729b2" } new_table_name: "default.Table_230a1a13b4584a188a4235462025db5e" modify_external_catalogs: true
W20240503 02:12:22.267078 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 24737f8a8ca74996b9f54d49d12b9288 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24737f8a8ca74996b9f54d49d12b9288 already in progress: creating tablet
I20240503 02:12:22.267812 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_dfe5a5a3b0c2426c912738f842cd96bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.268308 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfe5a5a3b0c2426c912738f842cd96bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.270927 4271 catalog_manager.cc:5484] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:22.271793 5114 ts_tablet_manager.cc:1888] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:22.271939 5114 log.cc:1196] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:22.272343 5114 ts_tablet_manager.cc:1909] T 9f57f3833a9240c88e0de1b486fae265 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:22.273008 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adc5b950ff6b4d9e8f02719738c475c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.280707 5103 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.281170 5103 ts_tablet_manager.cc:1367] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.003s
I20240503 02:12:22.281980 5103 raft_consensus.cc:348] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.285797 5103 raft_consensus.cc:374] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.285919 5103 raft_consensus.cc:729] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:22.286459 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64cdb861bc9f448698e243718136c303 consensus state has no leader
I20240503 02:12:22.286365 5103 consensus_queue.cc:260] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.292680 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_acbebd20514e4b8291c208833d606eb1" } new_table_name: "default.Table_f386f3dc954a401c9210cc2be3307c4d" modify_external_catalogs: true
I20240503 02:12:22.295540 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0ad200f18fbc4fa3883ee56709b508ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.296140 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ad200f18fbc4fa3883ee56709b508ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.297077 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 8750b02c5936412db525d2d3e77befeb from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.298564 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8fbb7708f7674d1f86c2b5e38808225b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.299105 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
W20240503 02:12:22.299204 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fbb7708f7674d1f86c2b5e38808225b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.300527 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cc2e1a713cc449a68a25796a8b26205a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.301077 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc2e1a713cc449a68a25796a8b26205a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.301900 5103 ts_tablet_manager.cc:1398] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240503 02:12:22.303869 5103 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.307579 5159 ts_tablet_manager.cc:1888] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.307780 5159 log.cc:1196] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:22.308676 5159 ts_tablet_manager.cc:1909] T 9f57f3833a9240c88e0de1b486fae265 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:22.312942 5159 tablet_replica.cc:330] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:22.313416 5159 raft_consensus.cc:2229] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.313623 5159 raft_consensus.cc:2258] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.314224 5159 ts_tablet_manager.cc:1875] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.312583 5103 tablet_bootstrap.cc:654] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.323935 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f0eecd86b1b2414f927cd22a6195fff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.324151 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_60dbaef44cae45268ac344822a93bbfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.324651 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0eecd86b1b2414f927cd22a6195fff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.324828 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e0cba6fcfb4c468dbf573bd12deb9897" } new_table_name: "default.Table_74b76179489146ee9c1af791b5976abe" modify_external_catalogs: true
I20240503 02:12:22.326655 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_36b7e35193f8454d896de4043269a0f7" } new_table_name: "default.Table_76170de28bcc444ba72ed46fca8665ba" modify_external_catalogs: true
W20240503 02:12:22.327651 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60dbaef44cae45268ac344822a93bbfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.329330 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bbf1abd1706f4e85a126f2c232a55a8b" } new_table_name: "default.Table_dd49ed261dd44569888a7e6f0dab8c66" modify_external_catalogs: true
I20240503 02:12:22.330212 5105 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.330375 5105 ts_tablet_manager.cc:1367] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.019s
I20240503 02:12:22.330818 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6aa097db3d14a28992284c5876aba0f" } modify_external_catalogs: true
I20240503 02:12:22.330977 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6aa097db3d14a28992284c5876aba0f" } modify_external_catalogs: true
I20240503 02:12:22.331315 5105 raft_consensus.cc:348] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.331550 5105 raft_consensus.cc:374] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.331604 5159 ts_tablet_manager.cc:1888] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.331655 5105 raft_consensus.cc:729] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.331894 5105 consensus_queue.cc:260] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.332509 5105 ts_tablet_manager.cc:1398] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:22.332903 5105 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:22.333132 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b38d63d2cc8c4f54ba43cdf8cde1841f consensus state has no leader
I20240503 02:12:22.333688 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4e1bf750865941a9b815ad9f398cda31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.334242 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e1bf750865941a9b815ad9f398cda31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.334789 5105 tablet_bootstrap.cc:654] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.337364 5106 tablet_bootstrap.cc:492] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.337553 5106 ts_tablet_manager.cc:1367] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.087s user 0.018s sys 0.013s
I20240503 02:12:22.338459 5106 raft_consensus.cc:348] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.338690 5106 raft_consensus.cc:374] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.338788 5106 raft_consensus.cc:729] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.331750 5159 log.cc:1196] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d989197a5d6e42379df6d8add449a913
I20240503 02:12:22.339030 5106 consensus_queue.cc:260] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.339349 5319 raft_consensus.cc:482] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.339445 5159 ts_tablet_manager.cc:1909] T d989197a5d6e42379df6d8add449a913 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:22.339480 5319 raft_consensus.cc:504] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.340340 5319 leader_election.cc:288] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:22.342063 4608 leader_election.cc:341] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 851756dc1a5942919c85967f83eca29a
I20240503 02:12:22.342095 5103 tablet_bootstrap.cc:492] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.342289 5103 ts_tablet_manager.cc:1367] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.002s
W20240503 02:12:22.342278 4608 leader_election.cc:341] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 851756dc1a5942919c85967f83eca29a
I20240503 02:12:22.342388 4608 leader_election.cc:302] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.342990 5319 raft_consensus.cc:2726] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.343065 5103 raft_consensus.cc:348] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.343375 5103 raft_consensus.cc:374] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.343462 5103 raft_consensus.cc:729] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.343421 5106 ts_tablet_manager.cc:1398] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:12:22.343798 5106 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.343710 5103 consensus_queue.cc:260] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.345715 5106 tablet_bootstrap.cc:654] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.346796 5103 ts_tablet_manager.cc:1398] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:22.347862 5103 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.349903 5103 tablet_bootstrap.cc:654] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.350245 5159 tablet_replica.cc:330] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:22.350533 5159 raft_consensus.cc:2229] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.351667 5159 raft_consensus.cc:2258] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.352353 5159 ts_tablet_manager.cc:1875] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.357829 5106 tablet_bootstrap.cc:492] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.358023 5106 ts_tablet_manager.cc:1367] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240503 02:12:22.358862 5106 raft_consensus.cc:348] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.359087 5106 raft_consensus.cc:374] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.359160 5106 raft_consensus.cc:729] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.359395 5106 consensus_queue.cc:260] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.360074 5106 ts_tablet_manager.cc:1398] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:22.361210 5106 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.363715 5106 tablet_bootstrap.cc:654] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.366645 5105 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.366986 5105 ts_tablet_manager.cc:1367] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20240503 02:12:22.367892 5159 ts_tablet_manager.cc:1888] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.368554 5105 raft_consensus.cc:348] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.368969 5105 raft_consensus.cc:374] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.369097 5105 raft_consensus.cc:729] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.369441 5105 consensus_queue.cc:260] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.370899 5103 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.371127 5103 ts_tablet_manager.cc:1367] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240503 02:12:22.371814 5105 ts_tablet_manager.cc:1398] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:22.372359 5105 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.372220 5103 raft_consensus.cc:348] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.373235 5103 raft_consensus.cc:374] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.373374 5103 raft_consensus.cc:729] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.373711 5103 consensus_queue.cc:260] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.375301 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8750b02c5936412db525d2d3e77befeb
I20240503 02:12:22.375856 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 8750b02c5936412db525d2d3e77befeb deleted and replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa
I20240503 02:12:22.375970 5106 tablet_bootstrap.cc:492] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.376137 5106 ts_tablet_manager.cc:1367] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:12:22.376190 5103 ts_tablet_manager.cc:1398] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:22.377174 5105 tablet_bootstrap.cc:654] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.377450 5103 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.377875 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:22.378388 5106 raft_consensus.cc:348] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.378690 5106 raft_consensus.cc:374] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.378819 5106 raft_consensus.cc:729] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.379287 5103 tablet_bootstrap.cc:654] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.368100 5159 log.cc:1196] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:22.379873 5159 ts_tablet_manager.cc:1909] T 3bc9fe2302024e5baa74d32941d70d29 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:22.380412 5106 consensus_queue.cc:260] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.381268 5106 ts_tablet_manager.cc:1398] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:12:22.382004 5159 tablet_replica.cc:330] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:22.382306 5159 raft_consensus.cc:2229] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.382540 5159 raft_consensus.cc:2258] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.382833 5106 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.383257 5159 ts_tablet_manager.cc:1875] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.384546 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 396b48865e954d05b9ba8084a2e7f29b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.384554 5106 tablet_bootstrap.cc:654] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.393515 5103 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.393865 5103 ts_tablet_manager.cc:1367] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:12:22.394083 5105 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.394301 5105 ts_tablet_manager.cc:1367] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20240503 02:12:22.395154 5105 raft_consensus.cc:348] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.395452 5105 raft_consensus.cc:374] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.395579 5105 raft_consensus.cc:729] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.395629 5103 raft_consensus.cc:348] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.395943 5103 raft_consensus.cc:374] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.396072 5103 raft_consensus.cc:729] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.395983 5105 consensus_queue.cc:260] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.396188 5201 raft_consensus.cc:482] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.396317 5201 raft_consensus.cc:504] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.396943 5201 leader_election.cc:288] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:22.397732 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5e627d818245e0bdeb16531499f7d5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:22.398069 5268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:22.398492 5268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:22.398627 5268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:22.398854 5105 ts_tablet_manager.cc:1398] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:22.399003 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5e627d818245e0bdeb16531499f7d5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.399201 5105 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:22.399981 4746 leader_election.cc:341] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.401208 5105 tablet_bootstrap.cc:654] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.396674 5103 consensus_queue.cc:260] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.401981 4529 raft_consensus.cc:2454] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.402734 4746 leader_election.cc:302] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:22.402982 5256 raft_consensus.cc:482] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.403132 5106 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.403239 5201 raft_consensus.cc:2781] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.403311 5106 ts_tablet_manager.cc:1367] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.006s
I20240503 02:12:22.403321 5201 raft_consensus.cc:482] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.403414 5201 raft_consensus.cc:3037] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.404078 5106 raft_consensus.cc:348] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.404292 5106 raft_consensus.cc:374] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.404354 5106 raft_consensus.cc:729] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.404603 5106 consensus_queue.cc:260] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.403133 5256 raft_consensus.cc:504] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.406109 5256 leader_election.cc:288] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.406090 5201 raft_consensus.cc:504] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.406900 5201 leader_election.cc:288] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:22.407452 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet cecaf785cb7e4aa1b6bef8d572e30706 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:22.407685 4608 leader_election.cc:341] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:22.407552 5106 ts_tablet_manager.cc:1398] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:22.408732 5106 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:22.409135 4608 leader_election.cc:341] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:22.409291 4608 leader_election.cc:302] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.410462 5106 tablet_bootstrap.cc:654] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.411000 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5e627d818245e0bdeb16531499f7d5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:22.411245 4529 raft_consensus.cc:3037] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:22.412742 5268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:22.412889 5268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:22.413002 5268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:22.413102 5268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:22.414286 4529 raft_consensus.cc:2454] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:22.415053 4746 leader_election.cc:302] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.415494 5201 raft_consensus.cc:2781] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.415772 5201 raft_consensus.cc:686] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:22.416064 5201 consensus_queue.cc:237] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.417881 5105 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.417794 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5e627d818245e0bdeb16531499f7d5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:22.417889 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b1056f446ede4a8c9a87907ab91da286"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.418062 5105 ts_tablet_manager.cc:1367] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240503 02:12:22.418370 5256 raft_consensus.cc:2726] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:22.418556 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1056f446ede4a8c9a87907ab91da286 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.418804 5105 raft_consensus.cc:348] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.419019 5105 raft_consensus.cc:374] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.419085 5105 raft_consensus.cc:729] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.419075 5103 ts_tablet_manager.cc:1398] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:12:22.419342 5105 consensus_queue.cc:260] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.420121 5105 ts_tablet_manager.cc:1398] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:22.420238 5103 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.420439 5105 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.423789 5105 tablet_bootstrap.cc:654] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.425897 5106 tablet_bootstrap.cc:492] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.425916 5286 raft_consensus.cc:482] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.426095 5286 raft_consensus.cc:504] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:22.267483 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b91abaf51abb4c2ca65412c3771c1ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b91abaf51abb4c2ca65412c3771c1ef2 already in progress: creating tablet
I20240503 02:12:22.426805 5286 leader_election.cc:288] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.426849 5106 ts_tablet_manager.cc:1367] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240503 02:12:22.427722 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:22.428592 4462 leader_election.cc:341] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.428802 5106 raft_consensus.cc:348] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.429328 5106 raft_consensus.cc:374] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:22.426796 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 159287us
I20240503 02:12:22.429543 5159 ts_tablet_manager.cc:1888] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.429709 5159 log.cc:1196] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a295609170a8482f8373dbf817b1eea1
W20240503 02:12:22.429679 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 959d998ec4ec4a52a6031650f2da4691 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959d998ec4ec4a52a6031650f2da4691 already in progress: creating tablet
I20240503 02:12:22.429608 5106 raft_consensus.cc:729] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:22.429978 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22776c81d1a24d368abb68274cf32a0b already in progress: creating tablet
I20240503 02:12:22.430099 5159 ts_tablet_manager.cc:1909] T a295609170a8482f8373dbf817b1eea1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:22.430318 5106 consensus_queue.cc:260] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.430665 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.431025 5103 tablet_bootstrap.cc:654] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.431027 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca08877cd948e1b04736b54bc779db (DEFAULT_TABLE table=default.Table_919992b1410a4a6494af1ad6995d7b93 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.431483 5106 ts_tablet_manager.cc:1398] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:22.431541 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca08877cd948e1b04736b54bc779db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.431818 5106 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.433542 5106 tablet_bootstrap.cc:654] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.436869 5159 tablet_replica.cc:330] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:22.439043 5159 raft_consensus.cc:2229] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.439291 5159 raft_consensus.cc:2258] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.440088 5105 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.440366 5105 ts_tablet_manager.cc:1367] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240503 02:12:22.440531 5268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=35675
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:22.441259 5105 raft_consensus.cc:348] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.441675 5105 raft_consensus.cc:374] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.441820 5105 raft_consensus.cc:729] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.442093 5105 consensus_queue.cc:260] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.442715 5103 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.442739 5268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:22.442850 5105 ts_tablet_manager.cc:1398] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:22.443213 5105 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.443681 5103 ts_tablet_manager.cc:1367] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20240503 02:12:22.444780 5103 raft_consensus.cc:348] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.439946 5159 ts_tablet_manager.cc:1875] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.445355 5268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:22.445654 5256 raft_consensus.cc:482] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.445818 5256 raft_consensus.cc:504] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.446517 5256 leader_election.cc:288] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.447129 5106 tablet_bootstrap.cc:492] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.447486 5106 ts_tablet_manager.cc:1367] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 02:12:22.447499 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.448496 5106 raft_consensus.cc:348] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.448788 5106 raft_consensus.cc:374] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.449059 5105 tablet_bootstrap.cc:654] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.449178 5106 raft_consensus.cc:729] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:22.449683 4746 leader_election.cc:341] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.450546 5106 consensus_queue.cc:260] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.450700 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.451222 5106 ts_tablet_manager.cc:1398] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:22.451582 5106 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:22.452404 4673 raft_consensus.cc:3037] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.454782 5106 tablet_bootstrap.cc:654] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:22.455031 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 87713e899d2148308316d807a6d76921 (table default.Table_4702ebe6617c4aebb1264298076c6a4d [id=31669972ad634736b38958103d7b6c16], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 87713e899d2148308316d807a6d76921 consensus state has no leader
I20240503 02:12:22.456189 5332 consensus_queue.cc:1035] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.462219 5105 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.462599 5105 ts_tablet_manager.cc:1367] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
W20240503 02:12:22.463507 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41d44fe8f6334572996665bdec60e2e5 consensus state has no leader
I20240503 02:12:22.463886 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 29e5c40f1f6b40308eed26b64d047494 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.463755 5105 raft_consensus.cc:348] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.464088 5105 raft_consensus.cc:374] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.464185 5105 raft_consensus.cc:729] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.464550 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 6cd402c0eb9e4cf49dc722f2dd214e09 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.464545 5105 consensus_queue.cc:260] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.465337 5105 ts_tablet_manager.cc:1398] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:22.465185 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_45f329757bf64154809c2a5b327f549f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.465726 5105 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:22.465905 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45f329757bf64154809c2a5b327f549f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.467414 5105 tablet_bootstrap.cc:654] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:22.467442 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4972057fae45168756d7c28ddbe2bf consensus state has no leader
I20240503 02:12:22.467810 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bc9150faaae244e8bc1953025cc8c19f" } modify_external_catalogs: true
I20240503 02:12:22.468008 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bc9150faaae244e8bc1953025cc8c19f" } modify_external_catalogs: true
I20240503 02:12:22.469844 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fd0e6dfdbbc14c7599538eb05a30083c" } modify_external_catalogs: true
I20240503 02:12:22.470019 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fd0e6dfdbbc14c7599538eb05a30083c" } modify_external_catalogs: true
I20240503 02:12:22.472894 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9b099c5cc2d14c2ba0fad6df023e947c" } modify_external_catalogs: true
I20240503 02:12:22.473116 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9b099c5cc2d14c2ba0fad6df023e947c" } modify_external_catalogs: true
I20240503 02:12:22.473584 5331 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.473732 5105 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.473888 5105 ts_tablet_manager.cc:1367] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240503 02:12:22.473734 5331 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.476125 5105 raft_consensus.cc:348] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.476709 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca08877cd948e1b04736b54bc779db (DEFAULT_TABLE table=default.Table_919992b1410a4a6494af1ad6995d7b93 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.476754 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27e37ebd0f49cdba7653548acbe1ea (DEFAULT_TABLE table=default.Table_4cc509b619474ecd8fc3f145f028ce24 [id=c99932f82ee449a98a52fbb18dd35129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.476975 5105 raft_consensus.cc:374] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.477253 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27e37ebd0f49cdba7653548acbe1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.477434 5105 raft_consensus.cc:729] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.477455 5277 consensus_queue.cc:1035] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.477734 5105 consensus_queue.cc:260] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.477939 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca08877cd948e1b04736b54bc779db. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.478428 5366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:22.479938 5105 ts_tablet_manager.cc:1398] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:22.480270 5105 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.481237 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 43119471f91c4a1ba900f267c88db9a6 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.481681 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43119471f91c4a1ba900f267c88db9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.481738 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 (DEFAULT_TABLE table=default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.481956 5105 tablet_bootstrap.cc:654] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.482136 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b01907833d44f6a000a4de3a2a6e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.486187 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27e37ebd0f49cdba7653548acbe1ea (DEFAULT_TABLE table=default.Table_4cc509b619474ecd8fc3f145f028ce24 [id=c99932f82ee449a98a52fbb18dd35129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.486663 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27e37ebd0f49cdba7653548acbe1ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.486989 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d919b9262ac48cbb8b3f893918b5590 consensus state has no leader
I20240503 02:12:22.487461 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_18dc31f244e94d6ab3c576eaae5e408b" } modify_external_catalogs: true
I20240503 02:12:22.487648 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_18dc31f244e94d6ab3c576eaae5e408b" } modify_external_catalogs: true
I20240503 02:12:22.488101 5105 tablet_bootstrap.cc:492] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.488281 5105 ts_tablet_manager.cc:1367] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 02:12:22.489248 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 43119471f91c4a1ba900f267c88db9a6 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.489641 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43119471f91c4a1ba900f267c88db9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.490060 5105 raft_consensus.cc:348] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.490306 5105 raft_consensus.cc:374] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.490412 5105 raft_consensus.cc:729] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.490725 5105 consensus_queue.cc:260] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.493259 5331 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.493402 5105 ts_tablet_manager.cc:1398] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:12:22.493769 5105 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.493949 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 (DEFAULT_TABLE table=default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.494182 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 91798b4bc2f74b9289425531937a78ce (DEFAULT_TABLE table=default.Table_92359e32d7a04637bfb894a7f9e1fb42 [id=a401eebd3ac541e19b6f3f310135a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.494413 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b01907833d44f6a000a4de3a2a6e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.494616 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91798b4bc2f74b9289425531937a78ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.495566 5105 tablet_bootstrap.cc:654] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.498080 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 91798b4bc2f74b9289425531937a78ce (DEFAULT_TABLE table=default.Table_92359e32d7a04637bfb894a7f9e1fb42 [id=a401eebd3ac541e19b6f3f310135a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.498483 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91798b4bc2f74b9289425531937a78ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.500641 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a (DEFAULT_TABLE table=default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.501120 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035216c9475d4b59aa1ac81b285b4d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.501806 5105 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.501951 5105 ts_tablet_manager.cc:1367] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:22.501902 4790 tablet_service.cc:1467] Processing CreateTablet for tablet cd66f88a24b74f2c9b2050af081d4601 (DEFAULT_TABLE table=default.Table_36b1bbc5ec49474881c8c6eca0df22ba [id=2d5bfb965c844a199fa81636be8265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.502378 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f (DEFAULT_TABLE table=default.Table_acbebd20514e4b8291c208833d606eb1 [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.503321 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.503938 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66f88a24b74f2c9b2050af081d4601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.505599 4641 tablet_service.cc:1467] Processing CreateTablet for tablet cd66f88a24b74f2c9b2050af081d4601 (DEFAULT_TABLE table=default.Table_36b1bbc5ec49474881c8c6eca0df22ba [id=2d5bfb965c844a199fa81636be8265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.505973 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66f88a24b74f2c9b2050af081d4601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.506579 5256 raft_consensus.cc:482] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.506773 5256 raft_consensus.cc:504] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.507474 5256 leader_election.cc:288] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.509294 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38d63d2cc8c4f54ba43cdf8cde1841f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.509600 4746 leader_election.cc:341] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.513442 5105 raft_consensus.cc:348] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.513700 5105 raft_consensus.cc:374] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:22.514302 4608 leader_election.cc:341] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:22.515058 5105 raft_consensus.cc:729] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.515571 5105 consensus_queue.cc:260] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.517453 4638 tablet_service.cc:1467] Processing CreateTablet for tablet de4e96b4be094101bf74904b2d66b6c7 (DEFAULT_TABLE table=default.Table_c1c8837504ad4efd959e72fc2f52c010 [id=166d3e6f76994946a92d66855ec30544]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.517988 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4e96b4be094101bf74904b2d66b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.519148 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a (DEFAULT_TABLE table=default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.521514 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.521160 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4972057fae45168756d7c28ddbe2bf (DEFAULT_TABLE table=default.Table_36b7e35193f8454d896de4043269a0f7 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.522128 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035216c9475d4b59aa1ac81b285b4d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.522343 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4972057fae45168756d7c28ddbe2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.525171 5105 ts_tablet_manager.cc:1398] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 02:12:22.527863 5331 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.530817 4787 tablet_service.cc:1467] Processing CreateTablet for tablet de4e96b4be094101bf74904b2d66b6c7 (DEFAULT_TABLE table=default.Table_c1c8837504ad4efd959e72fc2f52c010 [id=166d3e6f76994946a92d66855ec30544]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.531250 5105 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.531328 5331 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.532014 5331 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.533242 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.535532 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 88b173d78ba74ede801b12cccbeef6b1
I20240503 02:12:22.536273 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c1c8837504ad4efd959e72fc2f52c010" } new_table_name: "default.Table_0f920eca3cce4751b3898c378a8efd18" modify_external_catalogs: true
I20240503 02:12:22.539001 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9709e189b5424f23a7663146d1a912ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.539667 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9709e189b5424f23a7663146d1a912ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.542332 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4e96b4be094101bf74904b2d66b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.543025 4647 tablet_service.cc:1467] Processing CreateTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f (DEFAULT_TABLE table=default.Table_acbebd20514e4b8291c208833d606eb1 [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.544750 5105 tablet_bootstrap.cc:654] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.545017 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4972057fae45168756d7c28ddbe2bf (DEFAULT_TABLE table=default.Table_36b7e35193f8454d896de4043269a0f7 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.545574 5256 raft_consensus.cc:482] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.545732 5256 raft_consensus.cc:504] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.545830 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 (DEFAULT_TABLE table=default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.546108 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38d63d2cc8c4f54ba43cdf8cde1841f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.546187 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4972057fae45168756d7c28ddbe2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.546453 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b173d78ba74ede801b12cccbeef6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.546500 5256 leader_election.cc:288] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.543865 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:22.549618 4746 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:22.550253 4608 leader_election.cc:341] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:22.550763 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.551918 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 (DEFAULT_TABLE table=default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.552409 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b173d78ba74ede801b12cccbeef6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.552865 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.448223 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:22.451380 4462 leader_election.cc:341] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.553953 4462 leader_election.cc:302] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:22.554158 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:22.554351 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 102971us
I20240503 02:12:22.459497 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca08877cd948e1b04736b54bc779db (DEFAULT_TABLE table=default.Table_919992b1410a4a6494af1ad6995d7b93 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.555317 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 8195767d80384a6c9a4690292f6776de (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.555686 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca08877cd948e1b04736b54bc779db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.555840 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195767d80384a6c9a4690292f6776de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.557049 5318 raft_consensus.cc:2726] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.478895 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 8c27e37ebd0f49cdba7653548acbe1ea (DEFAULT_TABLE table=default.Table_4cc509b619474ecd8fc3f145f028ce24 [id=c99932f82ee449a98a52fbb18dd35129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.558044 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c27e37ebd0f49cdba7653548acbe1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.559626 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 396b48865e954d05b9ba8084a2e7f29b
I20240503 02:12:22.560066 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 396b48865e954d05b9ba8084a2e7f29b deleted and replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74
I20240503 02:12:22.482102 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 (DEFAULT_TABLE table=default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.485255 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 43119471f91c4a1ba900f267c88db9a6 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.494665 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 91798b4bc2f74b9289425531937a78ce (DEFAULT_TABLE table=default.Table_92359e32d7a04637bfb894a7f9e1fb42 [id=a401eebd3ac541e19b6f3f310135a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.497071 4493 tablet_service.cc:1467] Processing CreateTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f (DEFAULT_TABLE table=default.Table_acbebd20514e4b8291c208833d606eb1 [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.499881 4507 tablet_service.cc:1467] Processing CreateTablet for tablet cd66f88a24b74f2c9b2050af081d4601 (DEFAULT_TABLE table=default.Table_36b1bbc5ec49474881c8c6eca0df22ba [id=2d5bfb965c844a199fa81636be8265a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.500955 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a (DEFAULT_TABLE table=default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.502807 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.518961 4495 tablet_service.cc:1467] Processing CreateTablet for tablet de4e96b4be094101bf74904b2d66b6c7 (DEFAULT_TABLE table=default.Table_c1c8837504ad4efd959e72fc2f52c010 [id=166d3e6f76994946a92d66855ec30544]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.523914 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.525627 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4972057fae45168756d7c28ddbe2bf (DEFAULT_TABLE table=default.Table_36b7e35193f8454d896de4043269a0f7 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.531539 5286 raft_consensus.cc:482] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.564158 5286 raft_consensus.cc:504] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.564831 5286 leader_election.cc:288] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.532752 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.547765 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 (DEFAULT_TABLE table=default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.548553 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.551270 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.566679 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_deade72f82eb4fd18e15b3360d826a9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.567427 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deade72f82eb4fd18e15b3360d826a9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.568943 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43119471f91c4a1ba900f267c88db9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.569514 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38d63d2cc8c4f54ba43cdf8cde1841f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.569808 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 8195767d80384a6c9a4690292f6776de (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.569962 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91798b4bc2f74b9289425531937a78ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.570360 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b01907833d44f6a000a4de3a2a6e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.571125 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.571549 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66f88a24b74f2c9b2050af081d4601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.572098 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035216c9475d4b59aa1ac81b285b4d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.572690 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4972057fae45168756d7c28ddbe2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.573374 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4e96b4be094101bf74904b2d66b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.573472 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:22.573812 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b173d78ba74ede801b12cccbeef6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.593152 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d719c58974442281c7d694e1ca485e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.593509 5114 tablet_replica.cc:330] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:22.593716 5114 raft_consensus.cc:2229] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.593932 5114 raft_consensus.cc:2258] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.594539 5256 raft_consensus.cc:482] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.594720 5256 raft_consensus.cc:504] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.594942 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195767d80384a6c9a4690292f6776de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.595435 5256 leader_election.cc:288] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.596132 5114 ts_tablet_manager.cc:1875] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.604015 5114 ts_tablet_manager.cc:1888] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:22.604254 5114 log.cc:1196] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/86d719c58974442281c7d694e1ca485e
I20240503 02:12:22.604838 5114 ts_tablet_manager.cc:1909] T 86d719c58974442281c7d694e1ca485e P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:22.608021 5256 raft_consensus.cc:482] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.608223 5256 raft_consensus.cc:504] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.610749 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d719c58974442281c7d694e1ca485e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.611073 5256 leader_election.cc:288] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.611230 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 (DEFAULT_TABLE table=default.Table_b94a5914ab6549d1a420331f4fa47659 [id=ffbb2c8938d54556bf18d766e0c901cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.611660 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b8db679ba24ac6a93be591c0513ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.612406 4608 leader_election.cc:302] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.612857 5256 raft_consensus.cc:2726] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:22.613413 4608 leader_election.cc:341] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:22.613656 4608 leader_election.cc:302] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.613893 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet cecaf785cb7e4aa1b6bef8d572e30706
I20240503 02:12:22.613979 5256 raft_consensus.cc:2726] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:22.614341 4608 leader_election.cc:341] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:22.614404 4273 catalog_manager.cc:6554] ReplaceTablet: tablet cecaf785cb7e4aa1b6bef8d572e30706 deleted and replaced by tablet b143c398ccbe42f09fa2e33f04f0d4d7
I20240503 02:12:22.614488 4608 leader_election.cc:302] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.614729 5256 raft_consensus.cc:2726] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.617082 5115 tablet_replica.cc:330] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:22.617301 5115 raft_consensus.cc:2229] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.617491 5115 raft_consensus.cc:2258] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.617955 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8e996d76be4dda876f4645df56c1ce (DEFAULT_TABLE table=default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba [id=12382a23327a42f8bf299f03fa08b582]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.618350 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8e996d76be4dda876f4645df56c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.619349 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8122f1bfb50244dca4a7836db5ef79bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.620011 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8122f1bfb50244dca4a7836db5ef79bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.621299 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 2f862891b43a4845b723bfdb9a7da591 (DEFAULT_TABLE table=default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.621757 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f862891b43a4845b723bfdb9a7da591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.622864 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8e996d76be4dda876f4645df56c1ce (DEFAULT_TABLE table=default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba [id=12382a23327a42f8bf299f03fa08b582]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.623292 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8e996d76be4dda876f4645df56c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.623488 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 2f862891b43a4845b723bfdb9a7da591 (DEFAULT_TABLE table=default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.623823 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f862891b43a4845b723bfdb9a7da591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.626039 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d (DEFAULT_TABLE table=default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.626526 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0494ee4bd746069bfaf39bfb46687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.629602 5115 ts_tablet_manager.cc:1875] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.629760 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d (DEFAULT_TABLE table=default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.630172 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0494ee4bd746069bfaf39bfb46687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.632995 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 (DEFAULT_TABLE table=default.Table_b94a5914ab6549d1a420331f4fa47659 [id=ffbb2c8938d54556bf18d766e0c901cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.634207 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 289300b955dd41c09d2fc348224bb9f4 (DEFAULT_TABLE table=default.Table_8b04b325acf94deea5af5e4a9660f7d5 [id=2817ff73927a4593bddef5408db5cc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.634402 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e8c98d5ea84179a91fdbea8657d6e6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:22.635949 4746 leader_election.cc:341] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.636109 4746 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:22.636515 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 73061.9us
I20240503 02:12:22.637262 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:22.639070 5331 raft_consensus.cc:2726] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.639261 5277 raft_consensus.cc:482] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.640674 4641 tablet_service.cc:1467] Processing CreateTablet for tablet e9f630bf6a5740758fa1f81477ec59d3 (DEFAULT_TABLE table=default.Table_d510e8f3fde248f29d85de95822d83af [id=48dba2f59b8d4023849a605c2346dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.641325 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e8c98d5ea84179a91fdbea8657d6e6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.641348 5277 raft_consensus.cc:504] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:22.640408 4608 leader_election.cc:341] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:22.642679 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f630bf6a5740758fa1f81477ec59d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.642907 4746 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.642958 4813 raft_consensus.cc:2454] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:22.643673 4746 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.643900 4497 tablet_service.cc:1467] Processing CreateTablet for tablet e9f630bf6a5740758fa1f81477ec59d3 (DEFAULT_TABLE table=default.Table_d510e8f3fde248f29d85de95822d83af [id=48dba2f59b8d4023849a605c2346dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.644393 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b8db679ba24ac6a93be591c0513ab0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.646907 4608 leader_election.cc:341] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: ab3c55f80a644cefa87b5769f15bb7f0
I20240503 02:12:22.648083 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced251fecca84866ab3ff15bd225e2b7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.649435 4810 raft_consensus.cc:2454] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:22.649767 5331 raft_consensus.cc:2726] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.650665 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.651314 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 79884b3e73904c708d0d48bca85351b7 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:22.652997 5277 leader_election.cc:288] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.653170 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c4e56102ddd84ad58342dbdfb695e622" } new_table_name: "default.Table_b417b8282cd14a9aa9bcffd17aab880c" modify_external_catalogs: true
I20240503 02:12:22.651691 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3c55f80a644cefa87b5769f15bb7f0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:22.655174 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.655843 5105 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:22.658715 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 289300b955dd41c09d2fc348224bb9f4 (DEFAULT_TABLE table=default.Table_8b04b325acf94deea5af5e4a9660f7d5 [id=2817ff73927a4593bddef5408db5cc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.658970 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:22.660744 4746 leader_election.cc:341] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.661420 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d719c58974442281c7d694e1ca485e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.661585 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 (DEFAULT_TABLE table=default.Table_b94a5914ab6549d1a420331f4fa47659 [id=ffbb2c8938d54556bf18d766e0c901cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.662503 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 289300b955dd41c09d2fc348224bb9f4 (DEFAULT_TABLE table=default.Table_8b04b325acf94deea5af5e4a9660f7d5 [id=2817ff73927a4593bddef5408db5cc49]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:22.664108 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:22.660063 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.664496 5103 raft_consensus.cc:374] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.665726 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f630bf6a5740758fa1f81477ec59d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.666031 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289300b955dd41c09d2fc348224bb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.668557 4654 tablet_service.cc:1467] Processing CreateTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 (DEFAULT_TABLE table=default.Table_9b099c5cc2d14c2ba0fad6df023e947c [id=ec6cee0c5c6c466682653bba626dc0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.668975 4648 tablet_service.cc:1467] Processing CreateTablet for tablet d9dfb4150c3c49f4b5d4e31396da4e8d (DEFAULT_TABLE table=default.Table_be9606643d6b471db6114211488d37e0 [id=e0fcd7485e8c4af88cdb91ddebddc2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.671550 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:22.673013 4462 leader_election.cc:341] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.676656 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 9f57f3833a9240c88e0de1b486fae265 (table default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]) successfully deleted
I20240503 02:12:22.676901 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet d989197a5d6e42379df6d8add449a913 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
I20240503 02:12:22.677089 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3bc9fe2302024e5baa74d32941d70d29 (table default.Table_422f8213fa2e4d269fc4499098056f64 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]) successfully deleted
I20240503 02:12:22.677906 4791 tablet_service.cc:1467] Processing CreateTablet for tablet d9dfb4150c3c49f4b5d4e31396da4e8d (DEFAULT_TABLE table=default.Table_be9606643d6b471db6114211488d37e0 [id=e0fcd7485e8c4af88cdb91ddebddc2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.678025 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 9f57f3833a9240c88e0de1b486fae265 (table default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]) successfully deleted
W20240503 02:12:22.678247 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 95681e67db1e4485876d530133e8a514 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95681e67db1e4485876d530133e8a514 already in progress: creating tablet
W20240503 02:12:22.678469 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 already in progress: creating tablet
W20240503 02:12:22.678663 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 24737f8a8ca74996b9f54d49d12b9288 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24737f8a8ca74996b9f54d49d12b9288 already in progress: creating tablet
W20240503 02:12:22.678951 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b91abaf51abb4c2ca65412c3771c1ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b91abaf51abb4c2ca65412c3771c1ef2 already in progress: creating tablet
W20240503 02:12:22.679143 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 959d998ec4ec4a52a6031650f2da4691 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959d998ec4ec4a52a6031650f2da4691 already in progress: creating tablet
W20240503 02:12:22.679365 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22776c81d1a24d368abb68274cf32a0b already in progress: creating tablet
I20240503 02:12:22.679162 4777 tablet_service.cc:1467] Processing CreateTablet for tablet c2efdbfa909c475d95e99cf24206608f (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:22.679610 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 035216c9475d4b59aa1ac81b285b4d8a because the tablet was not found. No further retry: Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a
W20240503 02:12:22.679824 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because the tablet was not found. No further retry: Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
W20240503 02:12:22.680011 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because the tablet was not found. No further retry: Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:22.680169 4462 leader_election.cc:302] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:22.680262 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:22.680651 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 53017352b8f34955b116bbfd32f04854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53017352b8f34955b116bbfd32f04854 already in progress: creating tablet
I20240503 02:12:22.681399 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289300b955dd41c09d2fc348224bb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.683228 5286 raft_consensus.cc:2781] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.683403 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f2f9b962c9264f87b798df33544cb15d
I20240503 02:12:22.685067 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 86d719c58974442281c7d694e1ca485e (table default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]) successfully deleted
W20240503 02:12:22.687068 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 035216c9475d4b59aa1ac81b285b4d8a because the tablet was not found. No further retry: Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a
W20240503 02:12:22.687736 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because the tablet was not found. No further retry: Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
W20240503 02:12:22.687968 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because the tablet was not found. No further retry: Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:22.691972 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b2768a724a5944449a276cbef0d23e9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.692533 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 8195767d80384a6c9a4690292f6776de (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.692569 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8e996d76be4dda876f4645df56c1ce (DEFAULT_TABLE table=default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba [id=12382a23327a42f8bf299f03fa08b582]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.692821 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b8db679ba24ac6a93be591c0513ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.692994 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 2f862891b43a4845b723bfdb9a7da591 (DEFAULT_TABLE table=default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.693329 4785 tablet_service.cc:1467] Processing CreateTablet for tablet e9f630bf6a5740758fa1f81477ec59d3 (DEFAULT_TABLE table=default.Table_d510e8f3fde248f29d85de95822d83af [id=48dba2f59b8d4023849a605c2346dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:22.693537 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2768a724a5944449a276cbef0d23e9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.693428 5105 ts_tablet_manager.cc:1367] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.131s user 0.018s sys 0.027s
I20240503 02:12:22.694029 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 (DEFAULT_TABLE table=default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29 [id=08bfe029472248d7b17aaa9a8ee63472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.694641 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 (DEFAULT_TABLE table=default.Table_27e6472c5d7746f59ee6399fa6db4d3b [id=74533d0ffe7340578b7b2201d27f7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.695003 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.695152 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.693682 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 (DEFAULT_TABLE table=default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.693907 5277 raft_consensus.cc:482] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.693856 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d (DEFAULT_TABLE table=default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.695438 5277 raft_consensus.cc:504] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.695583 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 with delete_type TABLET_DATA_DELETED (replaced by tablet b143c398ccbe42f09fa2e33f04f0d4d7) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.695833 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.695035 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.696143 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dfb4150c3c49f4b5d4e31396da4e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.696169 5277 leader_election.cc:288] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.696442 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce11c70a924b8d92d8f36fc7401dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.696826 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e49cde2fd4aab9f44921d3c3921a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.697662 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8e996d76be4dda876f4645df56c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.701102 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2efdbfa909c475d95e99cf24206608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.701637 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d6d3ca80c54c629a5de82b06a7aac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.702718 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289300b955dd41c09d2fc348224bb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.705092 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f862891b43a4845b723bfdb9a7da591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.705106 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.705060 4782 tablet_service.cc:1467] Processing CreateTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 (DEFAULT_TABLE table=default.Table_9b099c5cc2d14c2ba0fad6df023e947c [id=ec6cee0c5c6c466682653bba626dc0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.705601 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f39e8d17b44895bd5c979d82145ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.709179 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195767d80384a6c9a4690292f6776de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.709359 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f630bf6a5740758fa1f81477ec59d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.713060 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9994ca5aea48d3bc25c097dddba3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.713019 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.717056 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0494ee4bd746069bfaf39bfb46687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.724858 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eec2c9a5654a4a8d6c3011ebc3c67c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.728224 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.729005 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eec2c9a5654a4a8d6c3011ebc3c67c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.729236 4673 raft_consensus.cc:2454] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.730120 4746 leader_election.cc:302] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:22.730609 5277 raft_consensus.cc:2781] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.730690 5103 raft_consensus.cc:729] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.731153 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f39e8d17b44895bd5c979d82145ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.731094 5277 raft_consensus.cc:482] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.731252 5277 raft_consensus.cc:3037] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.731468 4491 tablet_service.cc:1467] Processing CreateTablet for tablet c2efdbfa909c475d95e99cf24206608f (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.732594 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 (DEFAULT_TABLE table=default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29 [id=08bfe029472248d7b17aaa9a8ee63472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.733213 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b080fc5ffc564420a0b7f2b7460ebbbb
I20240503 02:12:22.734019 4608 leader_election.cc:302] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.734417 4637 tablet_service.cc:1467] Processing CreateTablet for tablet c2efdbfa909c475d95e99cf24206608f (DEFAULT_TABLE table=default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.732471 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 (DEFAULT_TABLE table=default.Table_27e6472c5d7746f59ee6399fa6db4d3b [id=74533d0ffe7340578b7b2201d27f7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.734962 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:22.735931 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9a928f9047e94439b5d83771a9629ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.736248 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
W20240503 02:12:22.736555 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a928f9047e94439b5d83771a9629ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.738421 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ebf1737b3dc84cc3bad535a9fc03ccdd" } new_table_name: "default.Table_6a4f81fe56f84e4ab87ea63d67015ae2" modify_external_catalogs: true
I20240503 02:12:22.738592 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f146f7cce0b24d3087bfe0973e2b7ef0
I20240503 02:12:22.739310 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b1797fad51024c2b851da2600276c0b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.739886 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1797fad51024c2b851da2600276c0b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.740026 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_919992b1410a4a6494af1ad6995d7b93" } new_table_name: "default.Table_e8ca78c1991543479153a973a8346732" modify_external_catalogs: true
I20240503 02:12:22.740816 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4a4f248f67ce475a8a517266b83052ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:22.741454 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a4f248f67ce475a8a517266b83052ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:22.743317 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de4e96b4be094101bf74904b2d66b6c7 consensus state has no leader
I20240503 02:12:22.743671 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_76170de28bcc444ba72ed46fca8665ba" } new_table_name: "default.Table_8e1cb6d17d9242f9903e21c37cbb3f72" modify_external_catalogs: true
I20240503 02:12:22.744402 5106 tablet_bootstrap.cc:492] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:22.744601 5106 ts_tablet_manager.cc:1367] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.293s user 0.030s sys 0.038s
I20240503 02:12:22.745191 5268 server_base.cc:1034] running on GCE node
I20240503 02:12:22.745793 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce11c70a924b8d92d8f36fc7401dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.747819 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3d67799b2e064626856a117b1c76c6d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.748497 5268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:22.748972 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.749811 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:22.752465 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f2f9b962c9264f87b798df33544cb15d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2f9b962c9264f87b798df33544cb15d already in progress: creating tablet
I20240503 02:12:22.752821 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9994ca5aea48d3bc25c097dddba3c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.753057 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9511e4de6c942b9b9e0790c88d3d543 consensus state has no leader
I20240503 02:12:22.753441 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf42415d1d604ff283f76a5ac98e7347" } modify_external_catalogs: true
I20240503 02:12:22.753630 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf42415d1d604ff283f76a5ac98e7347" } modify_external_catalogs: true
I20240503 02:12:22.752591 5277 raft_consensus.cc:504] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.755859 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2efdbfa909c475d95e99cf24206608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.755949 5103 consensus_queue.cc:260] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.756314 5318 raft_consensus.cc:482] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.757377 5318 raft_consensus.cc:504] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.758419 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 (DEFAULT_TABLE table=default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29 [id=08bfe029472248d7b17aaa9a8ee63472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.758570 5318 leader_election.cc:288] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.758860 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e49cde2fd4aab9f44921d3c3921a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.759122 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd (DEFAULT_TABLE table=default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.759670 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce11c70a924b8d92d8f36fc7401dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.760737 4505 tablet_service.cc:1467] Processing CreateTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 (DEFAULT_TABLE table=default.Table_9b099c5cc2d14c2ba0fad6df023e947c [id=ec6cee0c5c6c466682653bba626dc0c7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:22.761075 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d67799b2e064626856a117b1c76c6d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:22.763028 4608 leader_election.cc:341] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:22.763211 4608 leader_election.cc:302] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.765241 5268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:22.767485 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f39e8d17b44895bd5c979d82145ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.768690 5268 hybrid_clock.cc:648] HybridClock initialized: now 1714702342768542 us; error 135 us; skew 500 ppm
I20240503 02:12:22.769392 5268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240503 02:12:22.771332 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 396b48865e954d05b9ba8084a2e7f29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 396b48865e954d05b9ba8084a2e7f29b already in progress: creating tablet
I20240503 02:12:22.777096 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eec2c9a5654a4a8d6c3011ebc3c67c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:22.780763 5256 raft_consensus.cc:2781] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.780898 5256 raft_consensus.cc:482] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.781033 5256 raft_consensus.cc:3037] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:22.781409 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 45001.4us
W20240503 02:12:22.783485 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea4dac17dca3442d8934ac357779b78a consensus state has no leader
I20240503 02:12:22.783859 5256 raft_consensus.cc:504] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.784303 5319 raft_consensus.cc:482] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.784442 5319 raft_consensus.cc:504] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.785012 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d6d3ca80c54c629a5de82b06a7aac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.787029 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d989197a5d6e42379df6d8add449a913" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.789759 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 (DEFAULT_TABLE table=default.Table_27e6472c5d7746f59ee6399fa6db4d3b [id=74533d0ffe7340578b7b2201d27f7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.790237 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d6d3ca80c54c629a5de82b06a7aac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.791054 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.791323 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 with delete_type TABLET_DATA_DELETED (replaced by tablet b143c398ccbe42f09fa2e33f04f0d4d7) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.791785 5114 tablet_replica.cc:330] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:22.792026 5114 raft_consensus.cc:2229] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.792254 5114 raft_consensus.cc:2258] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:22.792321 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8750b02c5936412db525d2d3e77befeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8750b02c5936412db525d2d3e77befeb already in progress: creating tablet
I20240503 02:12:22.792464 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.793224 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 53b149e66a0446eeade190dd7f5d2bec (DEFAULT_TABLE table=default.Table_dfe5a5a3b0c2426c912738f842cd96bf [id=117dbc7727ae404e87952f3088016923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.793665 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b149e66a0446eeade190dd7f5d2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.795037 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.795295 4494 tablet_service.cc:1467] Processing CreateTablet for tablet d9dfb4150c3c49f4b5d4e31396da4e8d (DEFAULT_TABLE table=default.Table_be9606643d6b471db6114211488d37e0 [id=e0fcd7485e8c4af88cdb91ddebddc2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.795732 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dfb4150c3c49f4b5d4e31396da4e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.796730 5286 raft_consensus.cc:482] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.796892 5286 raft_consensus.cc:3037] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.801117 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 27c13f257eae44689af07cc31ea254c4 (DEFAULT_TABLE table=default.Table_adc5b950ff6b4d9e8f02719738c475c5 [id=ce02819e30324e95aa0ae9fadf8836f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.801754 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c13f257eae44689af07cc31ea254c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.802806 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.803164 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.803380 5103 ts_tablet_manager.cc:1398] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.322s user 0.003s sys 0.000s
I20240503 02:12:22.803828 5103 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.806185 5103 tablet_bootstrap.cc:654] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.807170 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:22.808575 4462 leader_election.cc:341] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d989197a5d6e42379df6d8add449a913
W20240503 02:12:22.809017 4462 leader_election.cc:341] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.809182 4462 leader_election.cc:302] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:22.809759 5318 raft_consensus.cc:2726] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.810562 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 1975c790bfa4449baa6d7b81509c5003 (DEFAULT_TABLE table=default.Table_0ad200f18fbc4fa3883ee56709b508ec [id=11645209f14b4d8880c3ca53263d4ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.811162 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1975c790bfa4449baa6d7b81509c5003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.814518 4526 raft_consensus.cc:2454] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.806622 4270 catalog_manager.cc:5484] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:22.778662 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.824872 5332 raft_consensus.cc:482] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.825950 5277 leader_election.cc:288] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.827420 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.827363 5332 raft_consensus.cc:504] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.829530 5332 leader_election.cc:288] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.829901 5201 raft_consensus.cc:482] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.829932 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:22.830145 4746 leader_election.cc:341] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.830102 5201 raft_consensus.cc:504] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.831176 5114 ts_tablet_manager.cc:1875] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:22.831175 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.831367 4746 leader_election.cc:302] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.833411 5201 leader_election.cc:288] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.834160 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.834204 5332 raft_consensus.cc:2726] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.836187 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.836596 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.837213 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.838291 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.839179 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 53b149e66a0446eeade190dd7f5d2bec (DEFAULT_TABLE table=default.Table_dfe5a5a3b0c2426c912738f842cd96bf [id=117dbc7727ae404e87952f3088016923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.839376 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 1975c790bfa4449baa6d7b81509c5003 (DEFAULT_TABLE table=default.Table_0ad200f18fbc4fa3883ee56709b508ec [id=11645209f14b4d8880c3ca53263d4ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.840490 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 27c13f257eae44689af07cc31ea254c4 (DEFAULT_TABLE table=default.Table_adc5b950ff6b4d9e8f02719738c475c5 [id=ce02819e30324e95aa0ae9fadf8836f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.842147 5332 raft_consensus.cc:482] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.842356 5332 raft_consensus.cc:504] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.843240 5332 leader_election.cc:288] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.845074 5115 ts_tablet_manager.cc:1888] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:22.846539 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eec2c9a5654a4a8d6c3011ebc3c67c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:22.846848 4673 raft_consensus.cc:3037] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.847680 5319 leader_election.cc:288] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.847879 5319 raft_consensus.cc:482] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.849083 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c07c7cbb743d4a8e9610281a00f94" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.848776 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:22.850701 4746 leader_election.cc:341] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:22.851331 4746 leader_election.cc:341] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.851439 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e42fa6e49c394853b67735ce31f70017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.851755 5256 leader_election.cc:288] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.851929 5256 raft_consensus.cc:482] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:22.852312 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e42fa6e49c394853b67735ce31f70017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.853017 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.853299 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_aeb4bc27f5dc47fca47f2316393e35c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:22.853474 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9994ca5aea48d3bc25c097dddba3c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:22.853840 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb4bc27f5dc47fca47f2316393e35c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:22.854379 5115 log.cc:1196] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/86d719c58974442281c7d694e1ca485e
I20240503 02:12:22.848047 5319 raft_consensus.cc:504] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.854965 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.856740 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.856884 5256 raft_consensus.cc:504] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.857360 5332 raft_consensus.cc:482] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.857594 5332 raft_consensus.cc:504] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.858580 5332 leader_election.cc:288] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.860033 4527 raft_consensus.cc:2379] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
W20240503 02:12:22.860324 4746 leader_election.cc:341] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.861107 4746 leader_election.cc:302] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.862221 4746 leader_election.cc:302] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.863080 5201 raft_consensus.cc:2726] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.863438 5201 raft_consensus.cc:3037] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:22.863677 4746 leader_election.cc:341] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:22.867686 5319 leader_election.cc:288] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.868484 5201 raft_consensus.cc:2726] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.872288 5115 ts_tablet_manager.cc:1909] T 86d719c58974442281c7d694e1ca485e P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:22.873971 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b149e66a0446eeade190dd7f5d2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.874357 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c13f257eae44689af07cc31ea254c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.874468 5286 raft_consensus.cc:504] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.876483 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1975c790bfa4449baa6d7b81509c5003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.881129 5286 leader_election.cc:288] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.888836 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b417b8282cd14a9aa9bcffd17aab880c" } modify_external_catalogs: true
I20240503 02:12:22.889204 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b417b8282cd14a9aa9bcffd17aab880c" } modify_external_catalogs: true
I20240503 02:12:22.891245 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c07c7cbb743d4a8e9610281a00f94" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.891772 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c9c3e0a1d400c94bfaefada2e9faa" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.892004 4671 raft_consensus.cc:2454] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.893708 5256 leader_election.cc:288] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:22.903040 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2efdbfa909c475d95e99cf24206608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.904953 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e8c98d5ea84179a91fdbea8657d6e6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:22.907634 4462 leader_election.cc:341] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:22.908385 4259 consensus_peers.cc:485] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d -> Peer 00c621d795c44d0daa534261920d874a (127.4.17.126:43321): Couldn't send request to peer 00c621d795c44d0daa534261920d874a. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.126:43321: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:12:22.908804 5201 raft_consensus.cc:482] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.909075 5201 raft_consensus.cc:504] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:22.909981 5201 leader_election.cc:288] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:22.911702 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e49cde2fd4aab9f44921d3c3921a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.913156 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dfb4150c3c49f4b5d4e31396da4e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.923794 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:22.926968 4673 raft_consensus.cc:2454] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:22.922753 5105 raft_consensus.cc:348] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.929010 5114 ts_tablet_manager.cc:1888] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:22.929162 5114 log.cc:1196] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/cecaf785cb7e4aa1b6bef8d572e30706
I20240503 02:12:22.928854 5105 raft_consensus.cc:374] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.929311 5105 raft_consensus.cc:729] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.929594 5114 ts_tablet_manager.cc:1909] T cecaf785cb7e4aa1b6bef8d572e30706 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:22.929632 5105 consensus_queue.cc:260] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.930279 5105 ts_tablet_manager.cc:1398] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.236s user 0.004s sys 0.000s
I20240503 02:12:22.930874 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]) successfully deleted
I20240503 02:12:22.936605 5159 ts_tablet_manager.cc:1888] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:22.936806 5159 log.cc:1196] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ab3c55f80a644cefa87b5769f15bb7f0
I20240503 02:12:22.937369 5159 ts_tablet_manager.cc:1909] T ab3c55f80a644cefa87b5769f15bb7f0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:22.938496 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 241) took 1047 ms. Trace:
I20240503 02:12:22.942487 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894b753b71a04193905a190da320a260" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:22.942899 4746 leader_election.cc:341] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.943681 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894b753b71a04193905a190da320a260" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:22.944497 4746 leader_election.cc:302] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.943384 5332 raft_consensus.cc:482] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.944878 5332 raft_consensus.cc:504] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.945055 4746 leader_election.cc:302] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.945535 4746 leader_election.cc:302] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.945709 5332 leader_election.cc:288] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:22.946192 5331 raft_consensus.cc:2726] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.946358 5331 raft_consensus.cc:2781] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.946460 5331 raft_consensus.cc:482] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.946547 5331 raft_consensus.cc:3037] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.946652 5201 raft_consensus.cc:2781] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.946986 5201 raft_consensus.cc:686] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:22.947439 5201 consensus_queue.cc:237] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.949329 5105 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:22.949478 5331 raft_consensus.cc:504] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.950163 5331 leader_election.cc:288] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:22.950796 4746 leader_election.cc:341] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:22.954237 4746 consensus_peers.cc:485] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d719c58974442281c7d694e1ca485e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:22.955276 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e8c98d5ea84179a91fdbea8657d6e6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:22.955534 4813 raft_consensus.cc:3037] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:22.956279 4746 leader_election.cc:341] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.956451 4746 leader_election.cc:302] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:22.956602 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:22.956902 4526 raft_consensus.cc:2454] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:22.957566 5277 raft_consensus.cc:2726] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.958343 4746 leader_election.cc:302] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.960578 4813 raft_consensus.cc:2454] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:22.962157 5277 raft_consensus.cc:2781] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.962342 5277 raft_consensus.cc:482] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.962509 5277 raft_consensus.cc:3037] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.965651 5277 raft_consensus.cc:504] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.966492 5277 leader_election.cc:288] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:22.966837 4746 leader_election.cc:341] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:22.967677 4462 leader_election.cc:302] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:22.968250 5286 raft_consensus.cc:2781] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.968748 5286 raft_consensus.cc:686] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:22.970224 5286 consensus_queue.cc:237] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:22.970757 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9f57f3833a9240c88e0de1b486fae265 because the tablet was not found. No further retry: Not found: Tablet not found: 9f57f3833a9240c88e0de1b486fae265
W20240503 02:12:22.971519 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66ca08877cd948e1b04736b54bc779db consensus state has no leader
W20240503 02:12:22.972357 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4972057fae45168756d7c28ddbe2bf consensus state has no leader
I20240503 02:12:22.973140 5103 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:22.973505 5103 ts_tablet_manager.cc:1367] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.170s user 0.000s sys 0.009s
W20240503 02:12:22.972589 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 88b173d78ba74ede801b12cccbeef6b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88b173d78ba74ede801b12cccbeef6b1 already in progress: creating tablet
I20240503 02:12:22.973861 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dfe5a5a3b0c2426c912738f842cd96bf" } modify_external_catalogs: true
I20240503 02:12:22.974049 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dfe5a5a3b0c2426c912738f842cd96bf" } modify_external_catalogs: true
I20240503 02:12:22.974390 5103 raft_consensus.cc:348] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.974633 5103 raft_consensus.cc:374] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.974710 5103 raft_consensus.cc:729] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:22.975001 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b080fc5ffc564420a0b7f2b7460ebbbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b080fc5ffc564420a0b7f2b7460ebbbb already in progress: creating tablet
I20240503 02:12:22.974938 5103 consensus_queue.cc:260] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.975520 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e45d748d1bce468aac5ffaa88cf90f15" } modify_external_catalogs: true
I20240503 02:12:22.975703 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e45d748d1bce468aac5ffaa88cf90f15" } modify_external_catalogs: true
I20240503 02:12:22.972707 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be53cf9a5c3a418d9dc24120c17d3a4b" } modify_external_catalogs: true
W20240503 02:12:22.977517 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c1f39e8d17b44895bd5c979d82145ad9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f39e8d17b44895bd5c979d82145ad9 already in progress: creating tablet
I20240503 02:12:22.977542 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be53cf9a5c3a418d9dc24120c17d3a4b" } modify_external_catalogs: true
W20240503 02:12:22.978802 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f146f7cce0b24d3087bfe0973e2b7ef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f146f7cce0b24d3087bfe0973e2b7ef0 already in progress: creating tablet
I20240503 02:12:22.981139 5103 ts_tablet_manager.cc:1398] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:22.982506 5103 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:22.983270 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:22.983515 4527 raft_consensus.cc:3037] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.986511 4527 raft_consensus.cc:2454] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:22.988013 4746 leader_election.cc:302] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:22.988379 5103 tablet_bootstrap.cc:654] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.990002 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:22.990983 5277 raft_consensus.cc:2781] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.991175 5277 raft_consensus.cc:686] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:22.991559 5277 consensus_queue.cc:237] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:22.996475 5105 tablet_bootstrap.cc:654] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.996402 5376 raft_consensus.cc:2726] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.998286 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b9511e4de6c942b9b9e0790c88d3d543
I20240503 02:12:22.998386 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 41ef3c6e5e7d40249b112144975ba220
I20240503 02:12:22.999158 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 79884b3e73904c708d0d48bca85351b7
I20240503 02:12:22.999404 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:22.999765 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 79884b3e73904c708d0d48bca85351b7 deleted and replaced by tablet cf10c48d165047668fbd5ca17f917fe3
I20240503 02:12:22.999641 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2fcdd1c688c74fe9848a495abd6ddde3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:23.000108 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.000118 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8a7d18f84dac48edb23e19370f9fdb8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.000274 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fcdd1c688c74fe9848a495abd6ddde3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:23.000629 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a7d18f84dac48edb23e19370f9fdb8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.000761 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:23.001513 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b9511e4de6c942b9b9e0790c88d3d543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9511e4de6c942b9b9e0790c88d3d543 already in progress: creating tablet
I20240503 02:12:23.002178 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c6a598ed8e5640d390928d3ab6fcbfde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.002637 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6a598ed8e5640d390928d3ab6fcbfde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:23.002797 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 41ef3c6e5e7d40249b112144975ba220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41ef3c6e5e7d40249b112144975ba220 already in progress: creating tablet
I20240503 02:12:23.003247 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_06ee7797a50e4886b1d001c0538b9d50" } new_table_name: "default.Table_634134e8f4a14d2a86f04be0d642a853" modify_external_catalogs: true
W20240503 02:12:23.003654 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 79884b3e73904c708d0d48bca85351b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79884b3e73904c708d0d48bca85351b7 already in progress: creating tablet
I20240503 02:12:23.004890 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:23.005255 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:23.007151 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet fe75d31e5da645be9ccdde4d0a4f9e05 (table default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f]). Expected version 1 got 0
I20240503 02:12:23.007298 5103 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:23.007442 5103 ts_tablet_manager.cc:1367] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 02:12:23.008348 5268 webserver.cc:460] Webserver started at http://127.4.17.126:35675/ using document root <none> and password file <none>
I20240503 02:12:23.008540 5103 raft_consensus.cc:348] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.008780 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.008874 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c9c3e0a1d400c94bfaefada2e9faa" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:23.007164 5106 raft_consensus.cc:348] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.009109 5106 raft_consensus.cc:374] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.009050 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_143457686a2e414c8d11b68e09860a08" } new_table_name: "default.Table_114d4d6efe8947578cff18d0c2f235d2" modify_external_catalogs: true
I20240503 02:12:23.009346 5114 tablet_replica.cc:330] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.009110 4673 raft_consensus.cc:3037] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.009572 5114 raft_consensus.cc:2229] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:23.009581 5268 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:23.009752 5268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:23.009829 5114 pending_rounds.cc:62] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:23.009891 5114 pending_rounds.cc:68] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420796808699904 op_type: NO_OP noop_request { }
I20240503 02:12:23.010020 5114 raft_consensus.cc:2866] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:23.010104 5114 raft_consensus.cc:2258] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.008781 5103 raft_consensus.cc:374] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.010316 5103 raft_consensus.cc:729] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.010679 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.010586 5103 consensus_queue.cc:260] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.011312 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.011474 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fdeb47dce7474e2fbaeeb3cb1201eb29" } new_table_name: "default.Table_755a293576f34174a553ccb8683695fe" modify_external_catalogs: true
I20240503 02:12:23.013294 5103 ts_tablet_manager.cc:1398] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:23.014189 5114 ts_tablet_manager.cc:1875] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.014441 5103 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:23.023468 5114 ts_tablet_manager.cc:1888] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:23.026826 5106 raft_consensus.cc:729] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.029881 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.030344 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.030616 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.030970 5268 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.009s sys 0.004s
W20240503 02:12:23.031141 4746 leader_election.cc:341] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:23.031797 4746 leader_election.cc:341] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.034729 5106 consensus_queue.cc:260] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.031472 5103 tablet_bootstrap.cc:654] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.035677 5114 log.cc:1196] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:23.038501 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7394c9d374404fb199ab6ff170d7febe
I20240503 02:12:23.041503 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0920c4cdbad14eaa91462804e2bdff73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.042052 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0920c4cdbad14eaa91462804e2bdff73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.042912 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 53b149e66a0446eeade190dd7f5d2bec
I20240503 02:12:23.043687 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_69bcfa1a666d49ecab96c811eec1b284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.044184 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69bcfa1a666d49ecab96c811eec1b284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.040637 4274 catalog_manager.cc:5484] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:23.045178 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0ea8c89cbb9e45d9af49c75e9921666e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.045631 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ea8c89cbb9e45d9af49c75e9921666e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.045647 5114 ts_tablet_manager.cc:1909] T 27e8c98d5ea84179a91fdbea8657d6e6 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:23.046610 5114 tablet_replica.cc:330] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.046792 5114 raft_consensus.cc:2229] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.046934 5114 raft_consensus.cc:2258] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.047441 5114 ts_tablet_manager.cc:1875] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:23.047539 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84b6e303e34a465caffed6c9e1e6bf74 has no consensus state
I20240503 02:12:23.048010 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d3b44d46eae4485e8605b99ad6fca885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.048574 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3b44d46eae4485e8605b99ad6fca885 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.049098 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet c0c18cecefab4b6caed0039d09cc1890 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:23.057610 5114 ts_tablet_manager.cc:1888] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:23.059626 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607e49cde2fd4aab9f44921d3c3921a5 consensus state has no leader
I20240503 02:12:23.059652 5277 consensus_queue.cc:1035] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:23.062376 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a90ebcf29004d1a925b69fa20998dd3 consensus state has no leader
I20240503 02:12:23.064971 5106 ts_tablet_manager.cc:1398] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.309s user 0.001s sys 0.002s
I20240503 02:12:23.065049 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 66ca08877cd948e1b04736b54bc779db from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:23.065346 5106 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.066642 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.066260 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.068493 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.068763 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.069326 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.071000 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet cecaf785cb7e4aa1b6bef8d572e30706 with delete_type TABLET_DATA_DELETED (replaced by tablet b143c398ccbe42f09fa2e33f04f0d4d7) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.071681 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.077167 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 53b149e66a0446eeade190dd7f5d2bec (DEFAULT_TABLE table=default.Table_dfe5a5a3b0c2426c912738f842cd96bf [id=117dbc7727ae404e87952f3088016923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.077632 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 27e8c98d5ea84179a91fdbea8657d6e6 (table default.Table_a7538f13504946b6af3145d6c6282670 [id=19642708612b48da91d318e497ba7580]) successfully deleted
I20240503 02:12:23.077950 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 1975c790bfa4449baa6d7b81509c5003 (DEFAULT_TABLE table=default.Table_0ad200f18fbc4fa3883ee56709b508ec [id=11645209f14b4d8880c3ca53263d4ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.938717 5159 rpcz_store.cc:276] 0503 02:12:21.891286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.892967 (+ 1681us) service_pool.cc:225] Handling call
0503 02:12:22.436763 (+543796us) ts_tablet_manager.cc:1120] Deleting tablet ab3c55f80a644cefa87b5769f15bb7f0
0503 02:12:22.436774 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:22.689135 (+252361us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:22.936409 (+247274us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:22.938423 (+ 2014us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":543614}
I20240503 02:12:23.080392 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 27c13f257eae44689af07cc31ea254c4 (DEFAULT_TABLE table=default.Table_adc5b950ff6b4d9e8f02719738c475c5 [id=ce02819e30324e95aa0ae9fadf8836f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.080451 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.080725 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ab3c55f80a644cefa87b5769f15bb7f0 (table default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]) successfully deleted
I20240503 02:12:23.081074 5159 tablet_replica.cc:330] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:23.081326 5159 raft_consensus.cc:2229] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.081538 5159 raft_consensus.cc:2258] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.081619 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.082166 5159 ts_tablet_manager.cc:1875] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.082865 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.083706 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1975c790bfa4449baa6d7b81509c5003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.084529 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 9327a9e5472b4eda854320c3a95f8e56 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:23.086135 4673 raft_consensus.cc:2454] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:23.086434 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.086854 4746 leader_election.cc:302] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.087256 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.087549 5277 raft_consensus.cc:2781] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:23.087693 5277 raft_consensus.cc:686] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:23.087966 5277 consensus_queue.cc:237] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.089107 5115 tablet_replica.cc:330] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.090559 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.090916 5106 tablet_bootstrap.cc:654] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.090991 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c13f257eae44689af07cc31ea254c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.092209 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 851756dc1a5942919c85967f83eca29a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.096850 5115 raft_consensus.cc:2229] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:23.097201 5115 raft_consensus.cc:2258] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.098065 5115 ts_tablet_manager.cc:1875] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.098337 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c9c3e0a1d400c94bfaefada2e9faa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.098661 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a829c48a5c24d6d9e6329ac34151d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.098963 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3c55f80a644cefa87b5769f15bb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.099007 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f72aa1636df46ce9eec9c69c2f70c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.099153 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc9fe2302024e5baa74d32941d70d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.099829 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eec2c9a5654a4a8d6c3011ebc3c67c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1aeba54a0e43d0b08aaf19bd5aa8d7) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.100620 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.103358 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced251fecca84866ab3ff15bd225e2b7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:23.103739 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c9c3e0a1d400c94bfaefada2e9faa" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:23.104418 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet ced251fecca84866ab3ff15bd225e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.104559 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fd4fb88436498999b5f339052ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.106860 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 159ea1eb32394b04b72258d24196202b
I20240503 02:12:23.107008 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.107455 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed525a61c734ee2b23d5d4f319a7d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.109022 4780 tablet_service.cc:1467] Processing CreateTablet for tablet c099a306a56643069f63e34a5f0d3dd7 (DEFAULT_TABLE table=default.Table_b1797fad51024c2b851da2600276c0b7 [id=bcb649eb00fc420caf6a643f4f30de02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.109463 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c099a306a56643069f63e34a5f0d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.111759 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 66ca08877cd948e1b04736b54bc779db
I20240503 02:12:23.112237 4273 catalog_manager.cc:6554] ReplaceTablet: tablet 66ca08877cd948e1b04736b54bc779db deleted and replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9
I20240503 02:12:23.113188 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9327a9e5472b4eda854320c3a95f8e56
I20240503 02:12:23.113729 4274 catalog_manager.cc:6554] ReplaceTablet: tablet 9327a9e5472b4eda854320c3a95f8e56 deleted and replaced by tablet 3478033c4ed94268a2436a2942c27a68
I20240503 02:12:23.114004 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b149e66a0446eeade190dd7f5d2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.114642 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:23.115145 4267 catalog_manager.cc:6554] ReplaceTablet: tablet c0c18cecefab4b6caed0039d09cc1890 deleted and replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e
I20240503 02:12:23.120903 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.121884 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57f3833a9240c88e0de1b486fae265 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.122212 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 546afc74e6c54f8aa5f7c684484d610b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.125959 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.126385 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043ba14eaa5b46319b0b88457d2a5f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.127970 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 1436dd3df7284f9bb379236eb9d76836 (DEFAULT_TABLE table=default.Table_b1056f446ede4a8c9a87907ab91da286 [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.128357 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1436dd3df7284f9bb379236eb9d76836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.128506 4777 tablet_service.cc:1467] Processing CreateTablet for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (DEFAULT_TABLE table=default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.128904 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b143c398ccbe42f09fa2e33f04f0d4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.131224 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 86d719c58974442281c7d694e1ca485e (table default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]) successfully deleted
W20240503 02:12:23.131511 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 53017352b8f34955b116bbfd32f04854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53017352b8f34955b116bbfd32f04854 already in progress: creating tablet
I20240503 02:12:23.132824 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 (DEFAULT_TABLE table=default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.132827 4783 tablet_service.cc:1467] Processing CreateTablet for tablet efa7c0c5fc614c16985008b8fba54bab (DEFAULT_TABLE table=default.Table_4e1bf750865941a9b815ad9f398cda31 [id=475bf67c3074494eb43e612a6897f57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.133317 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7c0c5fc614c16985008b8fba54bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.133564 4780 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d2c0991e849b5afa35e7230db298d (DEFAULT_TABLE table=default.Table_8122f1bfb50244dca4a7836db5ef79bb [id=a04586a0d80d4b0fa88bb0a6dc6b235f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.133886 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d2c0991e849b5afa35e7230db298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.137460 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b6e303e34a465caffed6c9e1e6bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.137434 5331 raft_consensus.cc:482] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.137585 5331 raft_consensus.cc:504] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.138299 5331 leader_election.cc:288] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.138823 5115 ts_tablet_manager.cc:1888] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:23.138983 5115 log.cc:1196] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/cecaf785cb7e4aa1b6bef8d572e30706
I20240503 02:12:23.138939 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c823155b774e42a5a422994903eae167" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:23.139454 5115 ts_tablet_manager.cc:1909] T cecaf785cb7e4aa1b6bef8d572e30706 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:23.139526 4746 leader_election.cc:341] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.067391 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.067976 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.146359 5114 log.cc:1196] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/22eec2c9a5654a4a8d6c3011ebc3c67c
I20240503 02:12:23.105937 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.146777 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed525a61c734ee2b23d5d4f319a7d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.146862 5114 ts_tablet_manager.cc:1909] T 22eec2c9a5654a4a8d6c3011ebc3c67c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:23.106338 4494 tablet_service.cc:1467] Processing CreateTablet for tablet c099a306a56643069f63e34a5f0d3dd7 (DEFAULT_TABLE table=default.Table_b1797fad51024c2b851da2600276c0b7 [id=bcb649eb00fc420caf6a643f4f30de02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.125516 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.126161 4505 tablet_service.cc:1467] Processing CreateTablet for tablet efa7c0c5fc614c16985008b8fba54bab (DEFAULT_TABLE table=default.Table_4e1bf750865941a9b815ad9f398cda31 [id=475bf67c3074494eb43e612a6897f57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.127146 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 1436dd3df7284f9bb379236eb9d76836 (DEFAULT_TABLE table=default.Table_b1056f446ede4a8c9a87907ab91da286 [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.128120 4499 tablet_service.cc:1467] Processing CreateTablet for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (DEFAULT_TABLE table=default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.124892 4504 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d2c0991e849b5afa35e7230db298d (DEFAULT_TABLE table=default.Table_8122f1bfb50244dca4a7836db5ef79bb [id=a04586a0d80d4b0fa88bb0a6dc6b235f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.139039 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c823155b774e42a5a422994903eae167" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:23.144102 4503 tablet_service.cc:1467] Processing CreateTablet for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (DEFAULT_TABLE table=default.Table_8fbb7708f7674d1f86c2b5e38808225b [id=45c3177a5f154c21a0be1e617e114785]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.130817 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 (DEFAULT_TABLE table=default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.145570 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd (DEFAULT_TABLE table=default.Table_cc2e1a713cc449a68a25796a8b26205a [id=2426f7ecae8b480789c148cbdffc5291]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.146692 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (DEFAULT_TABLE table=default.Table_04781eea28204bf196d694bd2986e194 [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.149649 5115 tablet_replica.cc:330] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.149868 5115 raft_consensus.cc:2229] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.150079 5115 raft_consensus.cc:2258] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.150732 5115 ts_tablet_manager.cc:1875] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:23.157229 4746 leader_election.cc:341] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.157707 4791 tablet_service.cc:1467] Processing CreateTablet for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (DEFAULT_TABLE table=default.Table_8fbb7708f7674d1f86c2b5e38808225b [id=45c3177a5f154c21a0be1e617e114785]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.158754 4270 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:23.159188 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf67d8c1d3ec4f758cca9745c42c7ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.159289 4746 leader_election.cc:302] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.160426 5331 raft_consensus.cc:2726] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.163100 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (DEFAULT_TABLE table=default.Table_04781eea28204bf196d694bd2986e194 [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.162863 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd (DEFAULT_TABLE table=default.Table_cc2e1a713cc449a68a25796a8b26205a [id=2426f7ecae8b480789c148cbdffc5291]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.163520 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.164839 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.165706 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367de4ff0a6f4abfa2924a070f387cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.166872 5115 ts_tablet_manager.cc:1888] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.167047 5115 log.cc:1196] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/851756dc1a5942919c85967f83eca29a
I20240503 02:12:23.167462 5115 ts_tablet_manager.cc:1909] T 851756dc1a5942919c85967f83eca29a P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.167558 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043ba14eaa5b46319b0b88457d2a5f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.169858 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 (DEFAULT_TABLE table=default.Table_9709e189b5424f23a7663146d1a912ac [id=5dfeefb7aef7484d8c5c9df2189f8932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.170274 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73965590a0ea4bffae6e99fa1367fad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.173067 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.173282 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.173305 5103 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:23.174073 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367de4ff0a6f4abfa2924a070f387cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.174144 5114 tablet_replica.cc:330] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.177718 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c099a306a56643069f63e34a5f0d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.178068 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7c0c5fc614c16985008b8fba54bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.178929 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1436dd3df7284f9bb379236eb9d76836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.180022 5115 tablet_replica.cc:330] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.180256 5115 raft_consensus.cc:2229] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.180408 5115 raft_consensus.cc:2258] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.181000 5115 ts_tablet_manager.cc:1875] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.182173 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d2c0991e849b5afa35e7230db298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.182302 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ceabc3a7225649c198519f5a34379f63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:23.182776 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf67d8c1d3ec4f758cca9745c42c7ee6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.182930 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceabc3a7225649c198519f5a34379f63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.184183 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 (DEFAULT_TABLE table=default.Table_9709e189b5424f23a7663146d1a912ac [id=5dfeefb7aef7484d8c5c9df2189f8932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.184711 4501 tablet_service.cc:1467] Processing CreateTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce (DEFAULT_TABLE table=default.Table_deade72f82eb4fd18e15b3360d826a9a [id=18e4f81f1509432ab8241d5d28e7a1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.184866 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_69bcfa1a666d49ecab96c811eec1b284" } new_table_name: "default.Table_048af81d4a0e49a2bd91a97f4765a567" modify_external_catalogs: true
I20240503 02:12:23.185024 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b143c398ccbe42f09fa2e33f04f0d4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.185528 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b6e303e34a465caffed6c9e1e6bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.185933 5103 ts_tablet_manager.cc:1367] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.160s user 0.017s sys 0.004s
I20240503 02:12:23.186116 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.186491 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed525a61c734ee2b23d5d4f319a7d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.186741 5103 raft_consensus.cc:348] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.187665 5106 tablet_bootstrap.cc:492] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.187826 5103 raft_consensus.cc:374] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.187953 5103 raft_consensus.cc:729] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.188202 5114 raft_consensus.cc:2229] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.188617 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet b38d63d2cc8c4f54ba43cdf8cde1841f from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:23.188776 5106 ts_tablet_manager.cc:1367] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.010s
I20240503 02:12:23.189551 5106 raft_consensus.cc:348] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.189924 5114 raft_consensus.cc:2258] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.190055 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c6a598ed8e5640d390928d3ab6fcbfde" } new_table_name: "default.Table_02a490be3c6544d5a09aa2551909cee3" modify_external_catalogs: true
I20240503 02:12:23.190527 5114 ts_tablet_manager.cc:1875] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.188468 5103 consensus_queue.cc:260] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.191215 5277 raft_consensus.cc:482] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.191455 5277 raft_consensus.cc:504] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.193265 5103 ts_tablet_manager.cc:1398] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:23.194504 5103 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:23.194869 5277 leader_election.cc:288] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.194897 4647 tablet_service.cc:1467] Processing CreateTablet for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (DEFAULT_TABLE table=default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.195089 5106 raft_consensus.cc:374] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.193320 4650 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d2c0991e849b5afa35e7230db298d (DEFAULT_TABLE table=default.Table_8122f1bfb50244dca4a7836db5ef79bb [id=a04586a0d80d4b0fa88bb0a6dc6b235f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.193820 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.195863 4652 tablet_service.cc:1467] Processing CreateTablet for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (DEFAULT_TABLE table=default.Table_8fbb7708f7674d1f86c2b5e38808225b [id=45c3177a5f154c21a0be1e617e114785]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.195955 4782 tablet_service.cc:1467] Processing CreateTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce (DEFAULT_TABLE table=default.Table_deade72f82eb4fd18e15b3360d826a9a [id=18e4f81f1509432ab8241d5d28e7a1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.196337 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfe6b2b9cab4517ae7cabec4796c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.196388 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd (DEFAULT_TABLE table=default.Table_cc2e1a713cc449a68a25796a8b26205a [id=2426f7ecae8b480789c148cbdffc5291]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.196548 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a596cfe654aa99e83d070b7884e6b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:23.194306 4649 tablet_service.cc:1467] Processing CreateTablet for tablet efa7c0c5fc614c16985008b8fba54bab (DEFAULT_TABLE table=default.Table_4e1bf750865941a9b815ad9f398cda31 [id=475bf67c3074494eb43e612a6897f57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.197000 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.194813 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 1436dd3df7284f9bb379236eb9d76836 (DEFAULT_TABLE table=default.Table_b1056f446ede4a8c9a87907ab91da286 [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.197264 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a596cfe654aa99e83d070b7884e6b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:23.197489 4527 raft_consensus.cc:2454] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:23.197664 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (DEFAULT_TABLE table=default.Table_04781eea28204bf196d694bd2986e194 [id=45383bf6b0004d8d8f059f46fbc0b4ee]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.198076 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dee34be89bda450296b916163bb10c46 consensus state has no leader
I20240503 02:12:23.198113 4746 leader_election.cc:302] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:23.198411 4746 leader_election.cc:341] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:23.198859 4608 leader_election.cc:341] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:23.192808 4641 tablet_service.cc:1467] Processing CreateTablet for tablet c099a306a56643069f63e34a5f0d3dd7 (DEFAULT_TABLE table=default.Table_b1797fad51024c2b851da2600276c0b7 [id=bcb649eb00fc420caf6a643f4f30de02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.195755 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 (DEFAULT_TABLE table=default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.199052 4638 tablet_service.cc:1467] Processing CreateTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce (DEFAULT_TABLE table=default.Table_deade72f82eb4fd18e15b3360d826a9a [id=18e4f81f1509432ab8241d5d28e7a1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.199200 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 (DEFAULT_TABLE table=default.Table_9709e189b5424f23a7663146d1a912ac [id=5dfeefb7aef7484d8c5c9df2189f8932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.195307 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b143c398ccbe42f09fa2e33f04f0d4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.200600 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73965590a0ea4bffae6e99fa1367fad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.200855 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043ba14eaa5b46319b0b88457d2a5f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.201037 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b6e303e34a465caffed6c9e1e6bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.201236 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1436dd3df7284f9bb379236eb9d76836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.201434 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367de4ff0a6f4abfa2924a070f387cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.201788 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf67d8c1d3ec4f758cca9745c42c7ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.202795 4778 tablet_service.cc:1467] Processing CreateTablet for tablet ec641e403ed843fda31dabd3920aeb68 (DEFAULT_TABLE table=default.Table_60dbaef44cae45268ac344822a93bbfd [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.203065 4502 tablet_service.cc:1467] Processing CreateTablet for tablet ec641e403ed843fda31dabd3920aeb68 (DEFAULT_TABLE table=default.Table_60dbaef44cae45268ac344822a93bbfd [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.203217 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec641e403ed843fda31dabd3920aeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.203429 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7c0c5fc614c16985008b8fba54bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.204417 4792 tablet_service.cc:1467] Processing CreateTablet for tablet a73ec6543b9a468492f3b97772ce43f9 (DEFAULT_TABLE table=default.Table_f0eecd86b1b2414f927cd22a6195fff2 [id=eb724fd2c11544ab9d8d539d30a2ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.204739 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73ec6543b9a468492f3b97772ce43f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.206087 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.206305 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.206491 4495 tablet_service.cc:1467] Processing CreateTablet for tablet a73ec6543b9a468492f3b97772ce43f9 (DEFAULT_TABLE table=default.Table_f0eecd86b1b2414f927cd22a6195fff2 [id=eb724fd2c11544ab9d8d539d30a2ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.206851 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73ec6543b9a468492f3b97772ce43f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.207085 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.207487 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfe6b2b9cab4517ae7cabec4796c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.208887 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.209453 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.211462 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.211488 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 88b173d78ba74ede801b12cccbeef6b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88b173d78ba74ede801b12cccbeef6b1 already in progress: creating tablet
W20240503 02:12:23.211745 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f2f9b962c9264f87b798df33544cb15d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2f9b962c9264f87b798df33544cb15d already in progress: creating tablet
W20240503 02:12:23.211995 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f146f7cce0b24d3087bfe0973e2b7ef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f146f7cce0b24d3087bfe0973e2b7ef0 already in progress: creating tablet
I20240503 02:12:23.212558 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]) successfully deleted
W20240503 02:12:23.212821 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8750b02c5936412db525d2d3e77befeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8750b02c5936412db525d2d3e77befeb already in progress: creating tablet
I20240503 02:12:23.213214 5277 raft_consensus.cc:2781] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:23.213416 5277 raft_consensus.cc:482] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:23.213614 5277 raft_consensus.cc:3037] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.220178 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:23.222704 5268 fs_manager.cc:725] Time spent opening block manager: real 0.189s user 0.009s sys 0.005s
I20240503 02:12:23.222999 5268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "00c621d795c44d0daa534261920d874a"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-ddxx"
I20240503 02:12:23.224603 5268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 02:12:23.226068 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 396b48865e954d05b9ba8084a2e7f29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 396b48865e954d05b9ba8084a2e7f29b already in progress: creating tablet
I20240503 02:12:23.228533 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.228986 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_02838524663542819d76fcab9d549504" } modify_external_catalogs: true
I20240503 02:12:23.229147 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_25ca6a7de9b2413ba4623e94f133fac8" } modify_external_catalogs: true
I20240503 02:12:23.229255 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_02838524663542819d76fcab9d549504" } modify_external_catalogs: true
I20240503 02:12:23.229300 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_25ca6a7de9b2413ba4623e94f133fac8" } modify_external_catalogs: true
W20240503 02:12:23.230551 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b080fc5ffc564420a0b7f2b7460ebbbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b080fc5ffc564420a0b7f2b7460ebbbb already in progress: creating tablet
W20240503 02:12:23.230896 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 79884b3e73904c708d0d48bca85351b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79884b3e73904c708d0d48bca85351b7 already in progress: creating tablet
I20240503 02:12:23.230887 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 94559cfc32704eae96b121f165686edc (DEFAULT_TABLE table=default.Table_0ea8c89cbb9e45d9af49c75e9921666e [id=6c3e164d6e3b4ca891ff750a862edbab]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.231262 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b9511e4de6c942b9b9e0790c88d3d543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9511e4de6c942b9b9e0790c88d3d543 already in progress: creating tablet
I20240503 02:12:23.231303 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d377db521b5346bbb8997345606a36e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.231670 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c1f39e8d17b44895bd5c979d82145ad9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f39e8d17b44895bd5c979d82145ad9 already in progress: creating tablet
W20240503 02:12:23.231920 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d377db521b5346bbb8997345606a36e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:23.232007 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 41ef3c6e5e7d40249b112144975ba220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41ef3c6e5e7d40249b112144975ba220 already in progress: creating tablet
I20240503 02:12:23.232188 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 851756dc1a5942919c85967f83eca29a (table default.Table_a38d73ea23ac44cab66f3cabba1f9fc2 [id=51b7109308df4de6994eb7cdd5751908]) successfully deleted
I20240503 02:12:23.232375 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (table default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]) successfully deleted
I20240503 02:12:23.233196 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d2c0991e849b5afa35e7230db298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.230791 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8af9d8455d574ede92363cd1cafb1334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.235244 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8af9d8455d574ede92363cd1cafb1334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.236373 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c099a306a56643069f63e34a5f0d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.241643 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.249449 5106 raft_consensus.cc:729] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.253502 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (DEFAULT_TABLE table=default.Table_b2768a724a5944449a276cbef0d23e9f [id=49901f1280764c1c8044fb4798a5c346]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.253696 5286 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.256261 5286 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.257038 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dacc67e3ff0c49fc9dddc714ec081269" } modify_external_catalogs: true
I20240503 02:12:23.257289 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dacc67e3ff0c49fc9dddc714ec081269" } modify_external_catalogs: true
I20240503 02:12:23.258077 5286 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:23.258267 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e6ff638542714113aa75b6b69e0de916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.258850 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6ff638542714113aa75b6b69e0de916 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.260195 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:23.260691 4273 catalog_manager.cc:6554] ReplaceTablet: tablet b38d63d2cc8c4f54ba43cdf8cde1841f deleted and replaced by tablet 54959b3e30944fab8bd5fd47ce42d246
I20240503 02:12:23.260571 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 (DEFAULT_TABLE table=default.Table_e42fa6e49c394853b67735ce31f70017 [id=60f853a8a91343cabaa48d3d929b1961]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.255553 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 42e80cc31f5c4908b82f2b030255f6af (DEFAULT_TABLE table=default.Table_d3b44d46eae4485e8605b99ad6fca885 [id=04d5d63e679c47ce9a72f1ffeeeaf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.261284 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c18cecefab4b6caed0039d09cc1890 with delete_type TABLET_DATA_DELETED (replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.262142 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94559cfc32704eae96b121f165686edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.262941 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_99049a1e00e84790bce035787060bd61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.263500 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99049a1e00e84790bce035787060bd61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.268379 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_34fcaf75826241bea9fc24cb331f823e" } modify_external_catalogs: true
I20240503 02:12:23.268575 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_34fcaf75826241bea9fc24cb331f823e" } modify_external_catalogs: true
I20240503 02:12:23.269663 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a1d5b2de8d4d3ba5190750545c9f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.270071 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f1c97fd1492d442e955a652306d0e6a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.270579 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c97fd1492d442e955a652306d0e6a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.270862 4490 tablet_service.cc:1467] Processing CreateTablet for tablet ee1fac803dd84eee8722f0a14559faf6 (DEFAULT_TABLE table=default.Table_9a928f9047e94439b5d83771a9629ffe [id=f7a7a1dbe706435a934d25f99b3114cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.273608 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc2ebfb11aa4840adcc35da73be75a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.275214 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3b41dce88919456aa846dbdcf5e2759f
I20240503 02:12:23.275592 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e80cc31f5c4908b82f2b030255f6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.276068 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_100da0be301548829b543438a8d63e02" } new_table_name: "default.Table_a6d935ff1f554a62a312091e6252d15a" modify_external_catalogs: true
I20240503 02:12:23.279009 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1fac803dd84eee8722f0a14559faf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.280961 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfe6b2b9cab4517ae7cabec4796c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.281701 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec641e403ed843fda31dabd3920aeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.282181 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 (DEFAULT_TABLE table=default.Table_e42fa6e49c394853b67735ce31f70017 [id=60f853a8a91343cabaa48d3d929b1961]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.282639 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc2ebfb11aa4840adcc35da73be75a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.282231 4787 tablet_service.cc:1467] Processing CreateTablet for tablet ee1fac803dd84eee8722f0a14559faf6 (DEFAULT_TABLE table=default.Table_9a928f9047e94439b5d83771a9629ffe [id=f7a7a1dbe706435a934d25f99b3114cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.283120 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73965590a0ea4bffae6e99fa1367fad0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.284466 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7efef931ca554ae18579bbea8d1c04f7 consensus state has no leader
I20240503 02:12:23.284492 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 42e80cc31f5c4908b82f2b030255f6af (DEFAULT_TABLE table=default.Table_d3b44d46eae4485e8605b99ad6fca885 [id=04d5d63e679c47ce9a72f1ffeeeaf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.284870 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_60dbaef44cae45268ac344822a93bbfd" } new_table_name: "default.Table_e6e97943c4dd41638a2b090765b119a3" modify_external_catalogs: true
I20240503 02:12:23.297040 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
I20240503 02:12:23.297930 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a95361cb35f04f1fb5b9e5ff04da58a1" } new_table_name: "default.Table_958a3d4a699a4387b0ff4c721a958d54" modify_external_catalogs: true
I20240503 02:12:23.302814 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ef5ce757a4e349cab1cdea6c1bc22736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.303320 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef5ce757a4e349cab1cdea6c1bc22736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.314163 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_799bd817690849dc9654a7fb51482f2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.314754 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_799bd817690849dc9654a7fb51482f2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.323819 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b161628065704c798020edbe098844c2
I20240503 02:12:23.324714 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_033e1f689d894383a2e2c0ebbbc6023b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.329983 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_033e1f689d894383a2e2c0ebbbc6023b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.335387 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cee118c15da24103b1ad3d2b2daf3f52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.335934 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cee118c15da24103b1ad3d2b2daf3f52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.337150 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2f862891b43a4845b723bfdb9a7da591
I20240503 02:12:23.337829 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_74b76179489146ee9c1af791b5976abe" } modify_external_catalogs: true
I20240503 02:12:23.338023 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_74b76179489146ee9c1af791b5976abe" } modify_external_catalogs: true
I20240503 02:12:23.338941 4504 tablet_service.cc:1467] Processing CreateTablet for tablet cf10c48d165047668fbd5ca17f917fe3 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.339493 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf10c48d165047668fbd5ca17f917fe3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.343712 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34eb69d5f4bd427a8e430a00fb076f83 consensus state has no leader
I20240503 02:12:23.344563 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c18cecefab4b6caed0039d09cc1890 with delete_type TABLET_DATA_DELETED (replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.355036 4491 tablet_service.cc:1467] Processing CreateTablet for tablet b743e99847bb47d1ae437c96789a8ea7 (DEFAULT_TABLE table=default.Table_3d67799b2e064626856a117b1c76c6d9 [id=c9dd0aca9dc74ed9bae21c1d8ee67678]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.362531 4499 tablet_service.cc:1467] Processing CreateTablet for tablet dee34be89bda450296b916163bb10c46 (DEFAULT_TABLE table=default.Table_048af81d4a0e49a2bd91a97f4765a567 [id=9f241c41ddaf480fa60902a73a61c36f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.363229 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec641e403ed843fda31dabd3920aeb68 consensus state has no leader
I20240503 02:12:23.363274 4508 tablet_service.cc:1467] Processing CreateTablet for tablet b84856917cb84c2a853c9a8a80246704 (DEFAULT_TABLE table=default.Table_8a7d18f84dac48edb23e19370f9fdb8f [id=83ba736efe7045f194563c021b282a5c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.365543 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c5d58f97b58481ea1dfa75397a32ae2 consensus state has no leader
I20240503 02:12:23.371666 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 7efef931ca554ae18579bbea8d1c04f7 (DEFAULT_TABLE table=default.Table_c6a598ed8e5640d390928d3ab6fcbfde [id=fde711a2c8e04377996a7cd87f629e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.390893 5268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:23.391363 4494 tablet_service.cc:1467] Processing CreateTablet for tablet c757a3a979c2489daa621cf8ea293ea8 (DEFAULT_TABLE table=default.Table_0920c4cdbad14eaa91462804e2bdff73 [id=ed2359e77c7c44208a625e7c4a672e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.392161 5268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:23.392570 5268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:23.394397 4503 tablet_service.cc:1467] Processing CreateTablet for tablet b32c377006a24372859ed44d59cf90ee (DEFAULT_TABLE table=default.Table_aeb4bc27f5dc47fca47f2316393e35c2 [id=921e31bf406040b9a44409fdfbb79d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.397255 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.259624 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:23.263659 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 46003.2us
W20240503 02:12:23.401228 4608 leader_election.cc:341] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: STOPPED
W20240503 02:12:23.401553 4608 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 8a829c48a5c24d6d9e6329ac34151d1f
W20240503 02:12:23.401727 4608 leader_election.cc:341] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:23.401841 4608 leader_election.cc:302] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.401914 5106 consensus_queue.cc:260] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:23.402248 4608 leader_election.cc:341] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:23.402249 5381 raft_consensus.cc:2726] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.402398 4608 leader_election.cc:302] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:23.402741 4608 leader_election.cc:341] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:23.402853 4608 leader_election.cc:302] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:23.402997 4608 raft_consensus.cc:2670] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:23.403256 4608 leader_election.cc:341] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:23.403366 4608 leader_election.cc:302] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.268112 5115 ts_tablet_manager.cc:1888] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.404845 5381 raft_consensus.cc:2726] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.405020 5115 log.cc:1196] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:23.405078 5381 raft_consensus.cc:2726] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.405458 5115 ts_tablet_manager.cc:1909] T 003c9c3e0a1d400c94bfaefada2e9faa P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.405639 5106 ts_tablet_manager.cc:1398] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.217s user 0.001s sys 0.001s
I20240503 02:12:23.405910 5115 tablet_replica.cc:330] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.406090 5115 raft_consensus.cc:2229] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.406270 5115 raft_consensus.cc:2258] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.406805 5115 ts_tablet_manager.cc:1875] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.407296 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 41d44fe8f6334572996665bdec60e2e5
I20240503 02:12:23.409020 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 94559cfc32704eae96b121f165686edc (DEFAULT_TABLE table=default.Table_0ea8c89cbb9e45d9af49c75e9921666e [id=6c3e164d6e3b4ca891ff750a862edbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.409066 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (DEFAULT_TABLE table=default.Table_b2768a724a5944449a276cbef0d23e9f [id=49901f1280764c1c8044fb4798a5c346]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.409341 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1fac803dd84eee8722f0a14559faf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.410233 4641 tablet_service.cc:1467] Processing CreateTablet for tablet ec641e403ed843fda31dabd3920aeb68 (DEFAULT_TABLE table=default.Table_60dbaef44cae45268ac344822a93bbfd [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.410610 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec641e403ed843fda31dabd3920aeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.412793 5106 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.412839 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:23.413096 4810 raft_consensus.cc:2454] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:23.413269 4777 tablet_service.cc:1467] Processing CreateTablet for tablet cf10c48d165047668fbd5ca17f917fe3 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.414340 5106 tablet_bootstrap.cc:654] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.419832 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.420600 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94559cfc32704eae96b121f165686edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.421039 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.422636 4784 tablet_service.cc:1467] Processing CreateTablet for tablet b743e99847bb47d1ae437c96789a8ea7 (DEFAULT_TABLE table=default.Table_3d67799b2e064626856a117b1c76c6d9 [id=c9dd0aca9dc74ed9bae21c1d8ee67678]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.423478 4462 leader_election.cc:341] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.424176 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c18cecefab4b6caed0039d09cc1890 with delete_type TABLET_DATA_DELETED (replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.424595 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.425086 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 42e80cc31f5c4908b82f2b030255f6af (DEFAULT_TABLE table=default.Table_d3b44d46eae4485e8605b99ad6fca885 [id=04d5d63e679c47ce9a72f1ffeeeaf506]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.425484 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e80cc31f5c4908b82f2b030255f6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.427664 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.427910 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.428089 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.428231 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.428172 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 94559cfc32704eae96b121f165686edc (DEFAULT_TABLE table=default.Table_0ea8c89cbb9e45d9af49c75e9921666e [id=6c3e164d6e3b4ca891ff750a862edbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.428963 4497 tablet_service.cc:1467] Processing CreateTablet for tablet d607da9689c14e51add9178a48b35913 (DEFAULT_TABLE table=default.Table_2fcdd1c688c74fe9848a495abd6ddde3 [id=9de9b4171d6e4a3583e141004418b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.429706 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (DEFAULT_TABLE table=default.Table_b2768a724a5944449a276cbef0d23e9f [id=49901f1280764c1c8044fb4798a5c346]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.430842 4637 tablet_service.cc:1467] Processing CreateTablet for tablet ee1fac803dd84eee8722f0a14559faf6 (DEFAULT_TABLE table=default.Table_9a928f9047e94439b5d83771a9629ffe [id=f7a7a1dbe706435a934d25f99b3114cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.431370 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.434167 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4ea17824609f4b25a85121fc6b203812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.434772 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea17824609f4b25a85121fc6b203812 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.436131 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2fcdd1c688c74fe9848a495abd6ddde3" } new_table_name: "default.Table_a9cfba004dfe4c7e8def7c54b105546f" modify_external_catalogs: true
I20240503 02:12:23.436968 4655 tablet_service.cc:1467] Processing CreateTablet for tablet b84856917cb84c2a853c9a8a80246704 (DEFAULT_TABLE table=default.Table_8a7d18f84dac48edb23e19370f9fdb8f [id=83ba736efe7045f194563c021b282a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.440311 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee34be89bda450296b916163bb10c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.442351 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efef931ca554ae18579bbea8d1c04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.443883 4638 tablet_service.cc:1467] Processing CreateTablet for tablet c757a3a979c2489daa621cf8ea293ea8 (DEFAULT_TABLE table=default.Table_0920c4cdbad14eaa91462804e2bdff73 [id=ed2359e77c7c44208a625e7c4a672e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.445483 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94559cfc32704eae96b121f165686edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.446213 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84856917cb84c2a853c9a8a80246704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.449069 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1fac803dd84eee8722f0a14559faf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.449132 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 (DEFAULT_TABLE table=default.Table_e42fa6e49c394853b67735ce31f70017 [id=60f853a8a91343cabaa48d3d929b1961]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.449527 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc2ebfb11aa4840adcc35da73be75a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.450356 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84856917cb84c2a853c9a8a80246704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.452980 4648 tablet_service.cc:1467] Processing CreateTablet for tablet cf10c48d165047668fbd5ca17f917fe3 (DEFAULT_TABLE table=default.Table_c2a0d32aa38141e8bd72a2e2293843c2 [id=156d16ce2a4246cd81968e098a9227dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.453361 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf10c48d165047668fbd5ca17f917fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.457867 4636 tablet_service.cc:1467] Processing CreateTablet for tablet b743e99847bb47d1ae437c96789a8ea7 (DEFAULT_TABLE table=default.Table_3d67799b2e064626856a117b1c76c6d9 [id=c9dd0aca9dc74ed9bae21c1d8ee67678]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.458243 4646 tablet_service.cc:1467] Processing CreateTablet for tablet dee34be89bda450296b916163bb10c46 (DEFAULT_TABLE table=default.Table_048af81d4a0e49a2bd91a97f4765a567 [id=9f241c41ddaf480fa60902a73a61c36f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.458547 4654 tablet_service.cc:1467] Processing CreateTablet for tablet b32c377006a24372859ed44d59cf90ee (DEFAULT_TABLE table=default.Table_aeb4bc27f5dc47fca47f2316393e35c2 [id=921e31bf406040b9a44409fdfbb79d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.458828 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c757a3a979c2489daa621cf8ea293ea8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.458897 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d607da9689c14e51add9178a48b35913 consensus state has no leader
I20240503 02:12:23.459534 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32c377006a24372859ed44d59cf90ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.460520 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743e99847bb47d1ae437c96789a8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.460597 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee34be89bda450296b916163bb10c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.461571 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743e99847bb47d1ae437c96789a8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.462034 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607da9689c14e51add9178a48b35913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.470650 5103 tablet_bootstrap.cc:654] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.471597 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c757a3a979c2489daa621cf8ea293ea8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.474787 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_02a490be3c6544d5a09aa2551909cee3 [id=fde711a2c8e04377996a7cd87f629e68], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7efef931ca554ae18579bbea8d1c04f7 consensus state has no leader
I20240503 02:12:23.474982 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a1d5b2de8d4d3ba5190750545c9f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.476382 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32c377006a24372859ed44d59cf90ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.482010 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8a7d18f84dac48edb23e19370f9fdb8f" } modify_external_catalogs: true
I20240503 02:12:23.482266 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8a7d18f84dac48edb23e19370f9fdb8f" } modify_external_catalogs: true
I20240503 02:12:23.488166 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_755a293576f34174a553ccb8683695fe" } new_table_name: "default.Table_1b852a3367de45db9bba22405e5e3bac" modify_external_catalogs: true
I20240503 02:12:23.493366 5277 raft_consensus.cc:504] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.496294 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e80cc31f5c4908b82f2b030255f6af. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:23.501256 5393 log.cc:924] Time spent T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
W20240503 02:12:23.509999 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 76001us
I20240503 02:12:23.512327 5114 ts_tablet_manager.cc:1888] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.512621 5114 log.cc:1196] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7d5e627d818245e0bdeb16531499f7d5
W20240503 02:12:23.513286 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 77282.6us
I20240503 02:12:23.518277 5332 consensus_queue.cc:1035] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.112s
I20240503 02:12:23.519325 4775 tablet_service.cc:1467] Processing CreateTablet for tablet c757a3a979c2489daa621cf8ea293ea8 (DEFAULT_TABLE table=default.Table_0920c4cdbad14eaa91462804e2bdff73 [id=ed2359e77c7c44208a625e7c4a672e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.520788 5105 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:23.520965 5105 ts_tablet_manager.cc:1367] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.572s user 0.009s sys 0.001s
I20240503 02:12:23.521672 5106 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.523375 5105 raft_consensus.cc:348] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.523626 5105 raft_consensus.cc:374] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.523737 5201 raft_consensus.cc:482] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.526301 5105 raft_consensus.cc:729] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.526619 5105 consensus_queue.cc:260] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.522320 4779 tablet_service.cc:1467] Processing CreateTablet for tablet b84856917cb84c2a853c9a8a80246704 (DEFAULT_TABLE table=default.Table_8a7d18f84dac48edb23e19370f9fdb8f [id=83ba736efe7045f194563c021b282a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.515697 4462 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:23.516007 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a596cfe654aa99e83d070b7884e6b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:23.532941 5105 ts_tablet_manager.cc:1398] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:23.534976 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743e99847bb47d1ae437c96789a8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.536355 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf10c48d165047668fbd5ca17f917fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.537819 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a1d5b2de8d4d3ba5190750545c9f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.537933 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.537842 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b32c377006a24372859ed44d59cf90ee (DEFAULT_TABLE table=default.Table_aeb4bc27f5dc47fca47f2316393e35c2 [id=921e31bf406040b9a44409fdfbb79d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.539330 4791 tablet_service.cc:1467] Processing CreateTablet for tablet d607da9689c14e51add9178a48b35913 (DEFAULT_TABLE table=default.Table_2fcdd1c688c74fe9848a495abd6ddde3 [id=9de9b4171d6e4a3583e141004418b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.539639 5277 leader_election.cc:288] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.540190 4641 tablet_service.cc:1467] Processing CreateTablet for tablet a73ec6543b9a468492f3b97772ce43f9 (DEFAULT_TABLE table=default.Table_f0eecd86b1b2414f927cd22a6195fff2 [id=eb724fd2c11544ab9d8d539d30a2ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.540887 4746 consensus_peers.cc:485] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d719c58974442281c7d694e1ca485e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:23.535809 4774 tablet_service.cc:1467] Processing CreateTablet for tablet dee34be89bda450296b916163bb10c46 (DEFAULT_TABLE table=default.Table_048af81d4a0e49a2bd91a97f4765a567 [id=9f241c41ddaf480fa60902a73a61c36f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.541426 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.541115 5106 ts_tablet_manager.cc:1367] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.000s
I20240503 02:12:23.543022 5382 raft_consensus.cc:482] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.543262 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c757a3a979c2489daa621cf8ea293ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.543376 5201 raft_consensus.cc:504] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.544699 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.543182 5382 raft_consensus.cc:504] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.546969 4636 tablet_service.cc:1467] Processing CreateTablet for tablet d607da9689c14e51add9178a48b35913 (DEFAULT_TABLE table=default.Table_2fcdd1c688c74fe9848a495abd6ddde3 [id=9de9b4171d6e4a3583e141004418b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.547868 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73ec6543b9a468492f3b97772ce43f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.550889 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a596cfe654aa99e83d070b7884e6b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:23.553968 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:23.556977 4746 leader_election.cc:341] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.562283 4527 raft_consensus.cc:3037] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.563386 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32c377006a24372859ed44d59cf90ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.563581 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 7efef931ca554ae18579bbea8d1c04f7 (DEFAULT_TABLE table=default.Table_02a490be3c6544d5a09aa2551909cee3 [id=fde711a2c8e04377996a7cd87f629e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.566259 5201 leader_election.cc:288] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.566345 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.568006 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aad7d6dcf2e4009872ce156c536f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.568101 5105 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:23.570288 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84856917cb84c2a853c9a8a80246704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.571606 5286 raft_consensus.cc:2781] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:23.571758 5286 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:23.572424 5286 raft_consensus.cc:3037] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.573436 5114 ts_tablet_manager.cc:1909] T 7d5e627d818245e0bdeb16531499f7d5 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:23.574184 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 52209.4us
I20240503 02:12:23.574987 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 54959b3e30944fab8bd5fd47ce42d246 (DEFAULT_TABLE table=default.Table_f386f3dc954a401c9210cc2be3307c4d [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.575520 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54959b3e30944fab8bd5fd47ce42d246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.576510 5159 ts_tablet_manager.cc:1888] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.576745 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 3478033c4ed94268a2436a2942c27a68 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.578485 4496 tablet_service.cc:1467] Processing CreateTablet for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (DEFAULT_TABLE table=default.Table_e8ca78c1991543479153a973a8346732 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.578954 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:23.580157 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3478033c4ed94268a2436a2942c27a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.583508 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607da9689c14e51add9178a48b35913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.584291 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5f8d331a718475a9f56dc7f3ec0e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.586719 5159 log.cc:1196] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ced251fecca84866ab3ff15bd225e2b7
I20240503 02:12:23.587962 5159 ts_tablet_manager.cc:1909] T ced251fecca84866ab3ff15bd225e2b7 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:23.592234 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:23.593176 4746 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.593417 4504 tablet_service.cc:1467] Processing CreateTablet for tablet aa67857b858c4719b498291c52b7aceb (DEFAULT_TABLE table=default.Table_ceabc3a7225649c198519f5a34379f63 [id=5c72e153c1034228b769b20591c2d596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.597445 5286 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.600817 4527 raft_consensus.cc:2454] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:23.601970 5286 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:23.609081 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.609935 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 235) took 1698 ms. Trace:
I20240503 02:12:23.611392 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa67857b858c4719b498291c52b7aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.612401 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.616660 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee34be89bda450296b916163bb10c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.617221 5159 rpcz_store.cc:276] 0503 02:12:21.890512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.891814 (+ 1302us) service_pool.cc:225] Handling call
0503 02:12:23.080992 (+1189178us) ts_tablet_manager.cc:1120] Deleting tablet ced251fecca84866ab3ff15bd225e2b7
0503 02:12:23.081005 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:23.215165 (+134160us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:23.576370 (+361205us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:23.588859 (+ 12489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183252}
I20240503 02:12:23.623895 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:23.624208 4810 raft_consensus.cc:3037] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.624629 5114 tablet_replica.cc:330] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.624846 5114 raft_consensus.cc:2229] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.625643 5114 raft_consensus.cc:2258] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.626277 5114 ts_tablet_manager.cc:1875] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.626397 4499 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb174825f1441f861ae6816137f855 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.629846 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb174825f1441f861ae6816137f855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.633355 4810 raft_consensus.cc:2454] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:23.638181 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.641212 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 7efef931ca554ae18579bbea8d1c04f7 (DEFAULT_TABLE table=default.Table_02a490be3c6544d5a09aa2551909cee3 [id=fde711a2c8e04377996a7cd87f629e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.642113 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.643151 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607da9689c14e51add9178a48b35913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.642905 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7d5e627d818245e0bdeb16531499f7d5 (table default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]) successfully deleted
I20240503 02:12:23.643536 4783 tablet_service.cc:1467] Processing CreateTablet for tablet aa67857b858c4719b498291c52b7aceb (DEFAULT_TABLE table=default.Table_ceabc3a7225649c198519f5a34379f63 [id=5c72e153c1034228b769b20591c2d596]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.643749 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7394c9d374404fb199ab6ff170d7febe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7394c9d374404fb199ab6ff170d7febe already in progress: creating tablet
W20240503 02:12:23.643988 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 53b149e66a0446eeade190dd7f5d2bec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b149e66a0446eeade190dd7f5d2bec already in progress: creating tablet
I20240503 02:12:23.644053 5201 raft_consensus.cc:482] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:23.644189 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 159ea1eb32394b04b72258d24196202b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea1eb32394b04b72258d24196202b already in progress: creating tablet
I20240503 02:12:23.644711 5201 raft_consensus.cc:504] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.644719 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:23.644408 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 66ca08877cd948e1b04736b54bc779db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66ca08877cd948e1b04736b54bc779db already in progress: creating tablet
W20240503 02:12:23.645768 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9327a9e5472b4eda854320c3a95f8e56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9327a9e5472b4eda854320c3a95f8e56 already in progress: creating tablet
W20240503 02:12:23.646041 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 27e8c98d5ea84179a91fdbea8657d6e6 because the tablet was not found. No further retry: Not found: Tablet not found: 27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:23.648583 5382 leader_election.cc:288] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.649056 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.650578 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:23.654939 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ced251fecca84866ab3ff15bd225e2b7 (table default.Table_4227e90304ee491ca298dabd17aa7b44 [id=536a3d22f6e34248b7936ba2e7a05fd5]) successfully deleted
I20240503 02:12:23.657107 5277 raft_consensus.cc:482] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.659741 5277 raft_consensus.cc:504] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.661660 5277 leader_election.cc:288] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.667868 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.671175 5201 leader_election.cc:288] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:23.676587 5115 ts_tablet_manager.cc:1888] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.676826 5115 log.cc:1196] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/8a829c48a5c24d6d9e6329ac34151d1f
I20240503 02:12:23.677572 5115 ts_tablet_manager.cc:1909] T 8a829c48a5c24d6d9e6329ac34151d1f P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.683135 5106 raft_consensus.cc:348] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.684274 4642 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb174825f1441f861ae6816137f855 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.684692 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.685003 4655 tablet_service.cc:1467] Processing CreateTablet for tablet aa67857b858c4719b498291c52b7aceb (DEFAULT_TABLE table=default.Table_ceabc3a7225649c198519f5a34379f63 [id=5c72e153c1034228b769b20591c2d596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.685328 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efef931ca554ae18579bbea8d1c04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.686301 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.689869 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa67857b858c4719b498291c52b7aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.690078 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aad7d6dcf2e4009872ce156c536f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.686659 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 3478033c4ed94268a2436a2942c27a68 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.693213 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3478033c4ed94268a2436a2942c27a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.686699 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 54959b3e30944fab8bd5fd47ce42d246 (DEFAULT_TABLE table=default.Table_f386f3dc954a401c9210cc2be3307c4d [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.696987 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb174825f1441f861ae6816137f855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.697268 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54959b3e30944fab8bd5fd47ce42d246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.701486 5106 raft_consensus.cc:374] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.701614 5106 raft_consensus.cc:729] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.701897 5106 consensus_queue.cc:260] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.707994 5106 ts_tablet_manager.cc:1398] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.166s user 0.002s sys 0.000s
I20240503 02:12:23.687479 4652 tablet_service.cc:1467] Processing CreateTablet for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (DEFAULT_TABLE table=default.Table_e8ca78c1991543479153a973a8346732 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.708379 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 003c9c3e0a1d400c94bfaefada2e9faa (table default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]) successfully deleted
I20240503 02:12:23.712158 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 8a829c48a5c24d6d9e6329ac34151d1f (table default.Table_4ab6febbc9724cf78cc642abf953ee61 [id=cad57d8578d2442984a701a12ebd16ca]) successfully deleted
I20240503 02:12:23.713224 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e8c98d5ea84179a91fdbea8657d6e6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:23.713726 4810 raft_consensus.cc:2379] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:23.713793 5382 consensus_queue.cc:1035] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.687877 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e8c98d5ea84179a91fdbea8657d6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.688947 5115 tablet_replica.cc:330] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.717608 5382 consensus_queue.cc:1035] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.721418 4790 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb174825f1441f861ae6816137f855 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.733405 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5f8d331a718475a9f56dc7f3ec0e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.733417 5106 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.721757 5159 tablet_replica.cc:330] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:23.735731 5159 raft_consensus.cc:2229] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:23.736140 5159 pending_rounds.cc:62] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:23.736218 5106 tablet_bootstrap.cc:654] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.736336 5159 pending_rounds.cc:68] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420796716273664 op_type: NO_OP noop_request { }
I20240503 02:12:23.736605 5159 raft_consensus.cc:2866] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:23.736774 5159 raft_consensus.cc:2258] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.721877 4746 leader_election.cc:302] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:23.737354 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 119169us
I20240503 02:12:23.738034 5159 ts_tablet_manager.cc:1875] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.738194 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:23.740286 4746 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.740387 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53dad4e45a184dc1ae4a82021a48e9be" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:23.740435 4746 leader_election.cc:302] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:23.740900 4746 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:23.741124 4746 leader_election.cc:341] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.722034 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efef931ca554ae18579bbea8d1c04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.742892 5115 raft_consensus.cc:2229] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.743131 5115 raft_consensus.cc:2258] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.743638 5426 raft_consensus.cc:2781] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:23.743886 5115 ts_tablet_manager.cc:1875] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.743897 5201 raft_consensus.cc:2726] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.744107 5426 raft_consensus.cc:686] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:23.744504 5426 consensus_queue.cc:237] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.748884 5159 ts_tablet_manager.cc:1888] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:23.749027 5106 tablet_bootstrap.cc:492] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.749215 5106 ts_tablet_manager.cc:1367] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 02:12:23.750149 5106 raft_consensus.cc:348] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.750371 5106 raft_consensus.cc:374] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.750483 5106 raft_consensus.cc:729] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.750672 5115 ts_tablet_manager.cc:1888] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.751217 5159 log.cc:1196] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/22eec2c9a5654a4a8d6c3011ebc3c67c
I20240503 02:12:23.752056 5159 ts_tablet_manager.cc:1909] T 22eec2c9a5654a4a8d6c3011ebc3c67c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:23.754151 5115 log.cc:1196] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/ab3c55f80a644cefa87b5769f15bb7f0
I20240503 02:12:23.755240 5115 ts_tablet_manager.cc:1909] T ab3c55f80a644cefa87b5769f15bb7f0 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.755241 5106 consensus_queue.cc:260] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.757001 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet ab3c55f80a644cefa87b5769f15bb7f0 (table default.Table_8c833315113b4fafb24834feb6fc6649 [id=0d2adf7b4fc6474d8795e4ca80906211]) successfully deleted
I20240503 02:12:23.756894 5115 tablet_replica.cc:330] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.758762 5115 raft_consensus.cc:2229] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.758939 5115 raft_consensus.cc:2258] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.759346 5105 tablet_bootstrap.cc:654] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.759464 5115 ts_tablet_manager.cc:1875] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.760824 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa67857b858c4719b498291c52b7aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.757434 5106 ts_tablet_manager.cc:1398] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:23.765429 5106 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.766860 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e (DEFAULT_TABLE table=default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.767448 5106 tablet_bootstrap.cc:654] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.767566 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aad7d6dcf2e4009872ce156c536f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.771112 5115 ts_tablet_manager.cc:1888] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.772723 5115 log.cc:1196] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/0f72aa1636df46ce9eec9c69c2f70c49
I20240503 02:12:23.773437 5115 ts_tablet_manager.cc:1909] T 0f72aa1636df46ce9eec9c69c2f70c49 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.773747 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 3478033c4ed94268a2436a2942c27a68 (DEFAULT_TABLE table=default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.774341 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3478033c4ed94268a2436a2942c27a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.775146 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 0f72aa1636df46ce9eec9c69c2f70c49 (table default.Table_f1c2ade4126d49df8aebe97256f2844f [id=efa3af2d2e444fdb9f4b3cbadb7271e8]) successfully deleted
I20240503 02:12:23.776613 5115 tablet_replica.cc:330] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.777098 5115 raft_consensus.cc:2229] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.777778 5115 raft_consensus.cc:2258] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.779096 5115 ts_tablet_manager.cc:1875] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.794910 5114 ts_tablet_manager.cc:1888] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.795110 5114 log.cc:1196] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/387a596cfe654aa99e83d070b7884e6b
I20240503 02:12:23.795480 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:23.795612 5114 ts_tablet_manager.cc:1909] T 387a596cfe654aa99e83d070b7884e6b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:23.795665 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53dad4e45a184dc1ae4a82021a48e9be" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:23.796571 5114 tablet_replica.cc:330] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.796756 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 387a596cfe654aa99e83d070b7884e6b (table default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]) successfully deleted
I20240503 02:12:23.796859 5114 raft_consensus.cc:2229] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.797044 5114 raft_consensus.cc:2258] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.806819 5103 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:23.807164 5103 ts_tablet_manager.cc:1367] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.613s user 0.032s sys 0.050s
W20240503 02:12:23.808558 4462 leader_election.cc:341] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:23.808712 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 157203us
I20240503 02:12:23.817016 4462 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:23.819700 5106 tablet_bootstrap.cc:492] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.819895 5106 ts_tablet_manager.cc:1367] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.000s
I20240503 02:12:23.820809 5106 raft_consensus.cc:348] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.821086 5106 raft_consensus.cc:374] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.821192 5106 raft_consensus.cc:729] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.821835 5106 consensus_queue.cc:260] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.826710 5106 ts_tablet_manager.cc:1398] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:23.827025 5114 ts_tablet_manager.cc:1875] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.828099 5286 raft_consensus.cc:2781] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:23.828874 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb174825f1441f861ae6816137f855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.829764 5103 raft_consensus.cc:348] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.829967 5103 raft_consensus.cc:374] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.830058 5103 raft_consensus.cc:729] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:23.830307 5331 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.036s sys 0.131s
I20240503 02:12:23.830309 5103 consensus_queue.cc:260] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:23.831058 4746 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:23.831450 4608 leader_election.cc:341] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:23.831575 4608 leader_election.cc:302] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.832000 5382 raft_consensus.cc:3037] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.833026 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d6e976e60b3e445492636b06856344c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.833752 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6e976e60b3e445492636b06856344c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.835642 5382 raft_consensus.cc:2726] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.835551 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3dce7a577f4f4969aa663ff5e3fb9234"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.836270 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dce7a577f4f4969aa663ff5e3fb9234 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.836429 5106 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.837905 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e8a9dc27e1654f2e92a1bd71999f7959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.838511 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8a9dc27e1654f2e92a1bd71999f7959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.838625 5106 tablet_bootstrap.cc:654] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.840354 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_adc5b950ff6b4d9e8f02719738c475c5" } modify_external_catalogs: true
I20240503 02:12:23.840584 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_adc5b950ff6b4d9e8f02719738c475c5" } modify_external_catalogs: true
I20240503 02:12:23.841528 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ceabc3a7225649c198519f5a34379f63" } modify_external_catalogs: true
I20240503 02:12:23.841571 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a6d935ff1f554a62a312091e6252d15a" } modify_external_catalogs: true
I20240503 02:12:23.841713 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ceabc3a7225649c198519f5a34379f63" } modify_external_catalogs: true
I20240503 02:12:23.841710 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a6d935ff1f554a62a312091e6252d15a" } modify_external_catalogs: true
I20240503 02:12:23.842450 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6e29e77c800340dbad41e6a6b3c1c0ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.843047 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e29e77c800340dbad41e6a6b3c1c0ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.844209 5115 ts_tablet_manager.cc:1888] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.844354 5115 log.cc:1196] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3bc9fe2302024e5baa74d32941d70d29
I20240503 02:12:23.844712 5115 ts_tablet_manager.cc:1909] T 3bc9fe2302024e5baa74d32941d70d29 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.844972 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d372fe582377472fb9b1ae98f230a96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:23.845645 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d372fe582377472fb9b1ae98f230a96f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:23.854588 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b84856917cb84c2a853c9a8a80246704
I20240503 02:12:23.848274 5382 raft_consensus.cc:482] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:23.855166 5382 raft_consensus.cc:504] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:23.855768 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet b84856917cb84c2a853c9a8a80246704 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.855877 5382 leader_election.cc:288] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:23.854907 5106 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.856736 5106 ts_tablet_manager.cc:1367] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 02:12:23.857779 5106 raft_consensus.cc:348] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.858062 5106 raft_consensus.cc:374] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.858151 5106 raft_consensus.cc:729] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.858461 5106 consensus_queue.cc:260] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.864502 5106 ts_tablet_manager.cc:1398] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:23.864948 5106 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:23.868044 5106 tablet_bootstrap.cc:654] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.870419 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 3bc9fe2302024e5baa74d32941d70d29 (table default.Table_422f8213fa2e4d269fc4499098056f64 [id=7ac5bd482a2c42ae9f7b67a3df4d5eb4]) successfully deleted
I20240503 02:12:23.870904 5115 tablet_replica.cc:330] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.871222 5115 raft_consensus.cc:2229] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.871523 5115 raft_consensus.cc:2258] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.872259 5115 ts_tablet_manager.cc:1875] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:23.877363 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607e49cde2fd4aab9f44921d3c3921a5 consensus state has no leader
I20240503 02:12:23.878739 5114 ts_tablet_manager.cc:1888] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:23.880254 4608 leader_election.cc:341] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 546afc74e6c54f8aa5f7c684484d610b
I20240503 02:12:23.882261 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet b84856917cb84c2a853c9a8a80246704 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:23.883424 5114 log.cc:1196] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/d989197a5d6e42379df6d8add449a913
I20240503 02:12:23.886171 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.887323 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5e627d818245e0bdeb16531499f7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.888078 5114 ts_tablet_manager.cc:1909] T d989197a5d6e42379df6d8add449a913 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:23.888310 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.889258 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet d989197a5d6e42379df6d8add449a913 with delete_type TABLET_DATA_DELETED (replaced by tablet ea7afb429d64497abcd9f1d0910651c3) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.890269 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f663b1d9cbdd47cb82ea64536b0e5013" } new_table_name: "default.Table_f471b5f3094048268179ff4ff91b119f" modify_external_catalogs: true
I20240503 02:12:23.892524 4746 leader_election.cc:302] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:23.893303 4170 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 98426.3us
W20240503 02:12:23.893766 5331 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.648s user 0.036s sys 0.131s
I20240503 02:12:23.893849 5106 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.893828 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet d989197a5d6e42379df6d8add449a913 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
I20240503 02:12:23.893985 5106 ts_tablet_manager.cc:1367] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.002s
W20240503 02:12:23.894109 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c0c18cecefab4b6caed0039d09cc1890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0c18cecefab4b6caed0039d09cc1890 already in progress: creating tablet
I20240503 02:12:23.894418 5114 tablet_replica.cc:330] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:23.894690 5114 raft_consensus.cc:2229] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.894984 5114 raft_consensus.cc:2258] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.894924 5106 raft_consensus.cc:348] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.895197 5106 raft_consensus.cc:374] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.895299 5106 raft_consensus.cc:729] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.895821 5114 ts_tablet_manager.cc:1875] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:23.898097 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 54959b3e30944fab8bd5fd47ce42d246 (DEFAULT_TABLE table=default.Table_f386f3dc954a401c9210cc2be3307c4d [id=aaefcb18de5b40f1b46a3ffe9bed832a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.904268 5115 ts_tablet_manager.cc:1888] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.904696 5106 consensus_queue.cc:260] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:23.906818 4608 leader_election.cc:341] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 546afc74e6c54f8aa5f7c684484d610b
I20240503 02:12:23.910281 4608 leader_election.cc:302] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:23.910913 5382 raft_consensus.cc:2726] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.911724 5286 raft_consensus.cc:686] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:23.912086 5286 consensus_queue.cc:237] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.914654 5106 ts_tablet_manager.cc:1398] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 02:12:23.915234 5103 ts_tablet_manager.cc:1398] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.105s user 0.010s sys 0.002s
I20240503 02:12:23.915678 5103 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:23.915910 5106 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:23.916193 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 87311.1us
W20240503 02:12:23.916532 4746 leader_election.cc:341] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:23.916700 4746 leader_election.cc:302] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:23.917198 4746 consensus_peers.cc:485] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d5e627d818245e0bdeb16531499f7d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:23.917742 5201 raft_consensus.cc:2726] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.918092 4672 raft_consensus.cc:3037] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.918589 4786 tablet_service.cc:1467] Processing CreateTablet for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (DEFAULT_TABLE table=default.Table_e8ca78c1991543479153a973a8346732 [id=997231149d3e427f824ed751b345d7b8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:23.922963 4746 consensus_peers.cc:485] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:23.923353 5277 consensus_queue.cc:1035] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.924950 5277 consensus_queue.cc:1035] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.939069 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet b84856917cb84c2a853c9a8a80246704 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:23.942778 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 240) took 1933 ms. Trace:
I20240503 02:12:23.943115 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54959b3e30944fab8bd5fd47ce42d246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.943733 5332 raft_consensus.cc:2726] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:23.947654 5106 tablet_bootstrap.cc:654] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.948424 5103 tablet_bootstrap.cc:654] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.909960 5115 log.cc:1196] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/22eec2c9a5654a4a8d6c3011ebc3c67c
I20240503 02:12:23.949513 5115 ts_tablet_manager.cc:1909] T 22eec2c9a5654a4a8d6c3011ebc3c67c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:23.952207 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (table default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]) successfully deleted
I20240503 02:12:23.952823 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:23.951532 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d3b44d46eae4485e8605b99ad6fca885" } new_table_name: "default.Table_b4d7bc3f9a7742dfa02441ebcb165e7b" modify_external_catalogs: true
I20240503 02:12:23.955078 5106 tablet_bootstrap.cc:492] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:23.964962 5114 ts_tablet_manager.cc:1888] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:23.965852 5114 log.cc:1196] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c823155b774e42a5a422994903eae167
I20240503 02:12:23.966706 5114 ts_tablet_manager.cc:1909] T c823155b774e42a5a422994903eae167 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:23.974087 5106 ts_tablet_manager.cc:1367] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
W20240503 02:12:23.974079 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7394c9d374404fb199ab6ff170d7febe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7394c9d374404fb199ab6ff170d7febe already in progress: creating tablet
I20240503 02:12:23.975000 5106 raft_consensus.cc:348] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:23.975260 5106 raft_consensus.cc:374] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.975329 5106 raft_consensus.cc:729] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.973776 5115 tablet_replica.cc:330] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:23.976852 5115 raft_consensus.cc:2229] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.977923 5115 raft_consensus.cc:2258] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.982707 5159 rpcz_store.cc:276] 0503 02:12:21.891180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.892794 (+ 1614us) service_pool.cc:225] Handling call
0503 02:12:23.617685 (+1724891us) ts_tablet_manager.cc:1120] Deleting tablet 22eec2c9a5654a4a8d6c3011ebc3c67c
0503 02:12:23.617702 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:23.745165 (+127463us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:23.748782 (+ 3617us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:23.825037 (+ 76255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1719826}
I20240503 02:12:23.984484 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5f8d331a718475a9f56dc7f3ec0e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.985275 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 22eec2c9a5654a4a8d6c3011ebc3c67c (table default.Table_eeccbe7767854540840b84af4d2820cf [id=b777bcefd1d44ea5868c0381aec37c2d]) successfully deleted
I20240503 02:12:23.994323 5105 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.003592 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 248) took 2054 ms. Trace:
I20240503 02:12:24.004633 5103 tablet_bootstrap.cc:492] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.005046 5103 ts_tablet_manager.cc:1367] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.005s
I20240503 02:12:24.006201 5103 raft_consensus.cc:348] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:24.007256 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet c823155b774e42a5a422994903eae167 (table default.Table_7f7a688514414e1cbb2d14173f3d1976 [id=7abc43f4f4eb42828defca24da288002]) successfully deleted
I20240503 02:12:24.008196 5382 raft_consensus.cc:482] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.008991 5382 raft_consensus.cc:504] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.009974 5382 leader_election.cc:288] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.010584 5115 ts_tablet_manager.cc:1875] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:24.013607 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3b41dce88919456aa846dbdcf5e2759f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b41dce88919456aa846dbdcf5e2759f already in progress: creating tablet
W20240503 02:12:24.014391 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ced4dfb2cf541af98d50d22d1a0f0ac already in progress: creating tablet
W20240503 02:12:24.015137 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2f862891b43a4845b723bfdb9a7da591 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f862891b43a4845b723bfdb9a7da591 already in progress: creating tablet
W20240503 02:12:24.015779 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 41d44fe8f6334572996665bdec60e2e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41d44fe8f6334572996665bdec60e2e5 already in progress: creating tablet
W20240503 02:12:24.016433 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b38d63d2cc8c4f54ba43cdf8cde1841f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38d63d2cc8c4f54ba43cdf8cde1841f already in progress: creating tablet
I20240503 02:12:24.011013 5106 consensus_queue.cc:260] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:24.017232 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b161628065704c798020edbe098844c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b161628065704c798020edbe098844c2 already in progress: creating tablet
I20240503 02:12:24.017428 5106 ts_tablet_manager.cc:1398] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
W20240503 02:12:24.018007 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b84856917cb84c2a853c9a8a80246704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84856917cb84c2a853c9a8a80246704 already in progress: creating tablet
I20240503 02:12:24.018445 5103 raft_consensus.cc:374] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.019148 5103 raft_consensus.cc:729] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.020076 5106 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.021533 5103 consensus_queue.cc:260] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:24.022794 5115 ts_tablet_manager.cc:1888] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.024693 5103 ts_tablet_manager.cc:1398] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 02:12:24.027287 5103 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.022927 5106 tablet_bootstrap.cc:654] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.033154 5115 log.cc:1196] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/ced251fecca84866ab3ff15bd225e2b7
W20240503 02:12:24.025406 4746 consensus_peers.cc:485] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 003c9c3e0a1d400c94bfaefada2e9faa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:24.035848 5115 ts_tablet_manager.cc:1909] T ced251fecca84866ab3ff15bd225e2b7 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.036186 5105 ts_tablet_manager.cc:1367] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.447s user 0.007s sys 0.006s
I20240503 02:12:24.036113 5159 rpcz_store.cc:276] 0503 02:12:21.932024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.932085 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:23.983038 (+2050953us) ts_tablet_manager.cc:1120] Deleting tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
0503 02:12:23.983050 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:23.986200 (+ 3150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2050740}
W20240503 02:12:24.038990 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 already in progress: creating tablet
I20240503 02:12:24.036561 5106 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.039933 5103 tablet_bootstrap.cc:654] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.050837 5103 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.051682 5103 ts_tablet_manager.cc:1367] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240503 02:12:24.039549 5106 ts_tablet_manager.cc:1367] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240503 02:12:24.054082 5115 tablet_replica.cc:330] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.055109 5115 raft_consensus.cc:2229] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.055380 5115 raft_consensus.cc:2258] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.054759 5106 raft_consensus.cc:348] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.055634 5106 raft_consensus.cc:374] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.055775 5106 raft_consensus.cc:729] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.056074 5106 consensus_queue.cc:260] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.056691 5106 ts_tablet_manager.cc:1398] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:24.057371 5115 ts_tablet_manager.cc:1875] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.057741 5106 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.062191 5106 tablet_bootstrap.cc:654] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.066061 5103 raft_consensus.cc:348] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.067057 5105 raft_consensus.cc:348] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.068120 5105 raft_consensus.cc:374] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.068253 5105 raft_consensus.cc:729] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.068614 5106 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.068789 5106 ts_tablet_manager.cc:1367] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:12:24.068840 5105 consensus_queue.cc:260] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.069609 5105 ts_tablet_manager.cc:1398] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240503 02:12:24.069600 5106 raft_consensus.cc:348] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.069821 5106 raft_consensus.cc:374] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.069891 5106 raft_consensus.cc:729] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.070115 5106 consensus_queue.cc:260] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.070626 5106 ts_tablet_manager.cc:1398] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:24.070775 5103 raft_consensus.cc:374] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.071166 5106 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.072911 5106 tablet_bootstrap.cc:654] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.073107 5103 raft_consensus.cc:729] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.073494 5103 consensus_queue.cc:260] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.079828 5103 ts_tablet_manager.cc:1398] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20240503 02:12:24.088271 5103 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.090610 5103 tablet_bootstrap.cc:654] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.076318 5115 ts_tablet_manager.cc:1888] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.091817 5115 log.cc:1196] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/14fd4fb88436498999b5f339052ac0b6
I20240503 02:12:24.092319 5115 ts_tablet_manager.cc:1909] T 14fd4fb88436498999b5f339052ac0b6 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.093639 5115 tablet_replica.cc:330] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.095016 5115 raft_consensus.cc:2229] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.095739 5115 raft_consensus.cc:2258] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.096858 5115 ts_tablet_manager.cc:1875] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.099000 5159 tablet_replica.cc:330] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.099237 5159 raft_consensus.cc:2229] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.099491 5159 raft_consensus.cc:2258] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.100279 5159 ts_tablet_manager.cc:1875] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.106703 5106 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.106884 5106 ts_tablet_manager.cc:1367] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20240503 02:12:24.107797 5106 raft_consensus.cc:348] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.108104 5106 raft_consensus.cc:374] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.108213 5106 raft_consensus.cc:729] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.108445 5106 consensus_queue.cc:260] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.109014 5106 ts_tablet_manager.cc:1398] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:24.110738 5105 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.113183 5105 tablet_bootstrap.cc:654] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.117873 5103 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.117874 5115 ts_tablet_manager.cc:1888] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.118047 5115 log.cc:1196] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/9f57f3833a9240c88e0de1b486fae265
I20240503 02:12:24.118439 5115 ts_tablet_manager.cc:1909] T 9f57f3833a9240c88e0de1b486fae265 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.118441 5103 ts_tablet_manager.cc:1367] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 02:12:24.118882 5106 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.119897 5103 raft_consensus.cc:348] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.120375 5159 ts_tablet_manager.cc:1888] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.120520 5159 log.cc:1196] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:24.120524 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 288) took 1020 ms. Trace:
I20240503 02:12:24.120630 5115 rpcz_store.cc:276] 0503 02:12:23.100314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.105458 (+ 5144us) service_pool.cc:225] Handling call
0503 02:12:24.093575 (+988117us) ts_tablet_manager.cc:1120] Deleting tablet 9f57f3833a9240c88e0de1b486fae265
0503 02:12:24.093582 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.106766 (+ 13184us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.117728 (+ 10962us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.120461 (+ 2733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971566}
I20240503 02:12:24.121122 5115 tablet_replica.cc:330] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.121345 5115 raft_consensus.cc:2229] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.121539 5103 raft_consensus.cc:374] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.122822 5106 tablet_bootstrap.cc:654] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.122982 5103 raft_consensus.cc:729] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.123518 5103 consensus_queue.cc:260] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.127038 5103 ts_tablet_manager.cc:1398] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:24.125361 5115 raft_consensus.cc:2258] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.127852 5115 ts_tablet_manager.cc:1875] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.129132 5105 tablet_bootstrap.cc:492] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.130144 5105 ts_tablet_manager.cc:1367] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240503 02:12:24.130640 5103 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.131422 5105 raft_consensus.cc:348] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.135022 5115 ts_tablet_manager.cc:1888] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.138794 5286 raft_consensus.cc:482] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.139026 5286 raft_consensus.cc:504] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.140059 5286 leader_election.cc:288] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.141147 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24737f8a8ca74996b9f54d49d12b9288" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.143180 4672 raft_consensus.cc:2454] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.143795 4462 leader_election.cc:302] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.144272 5286 raft_consensus.cc:2781] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.144425 5286 raft_consensus.cc:482] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.144539 5286 raft_consensus.cc:3037] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.146548 5103 tablet_bootstrap.cc:654] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.146607 5478 raft_consensus.cc:482] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.146754 5478 raft_consensus.cc:504] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.147846 5478 leader_election.cc:288] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.150183 5286 raft_consensus.cc:504] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.151718 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24737f8a8ca74996b9f54d49d12b9288" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:24.151932 4672 raft_consensus.cc:3037] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.154269 5103 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.155323 4672 raft_consensus.cc:2454] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:24.155468 5103 ts_tablet_manager.cc:1367] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240503 02:12:24.156477 5286 leader_election.cc:288] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.157063 4462 leader_election.cc:302] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.157315 5495 tablet.cc:1722] T fe75d31e5da645be9ccdde4d0a4f9e05 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:24.156818 5115 log.cc:1196] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/546afc74e6c54f8aa5f7c684484d610b
I20240503 02:12:24.159794 5115 ts_tablet_manager.cc:1909] T 546afc74e6c54f8aa5f7c684484d610b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.141731 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24737f8a8ca74996b9f54d49d12b9288" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.162807 5105 raft_consensus.cc:374] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.156608 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24737f8a8ca74996b9f54d49d12b9288" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.162920 5105 raft_consensus.cc:729] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.163465 5105 consensus_queue.cc:260] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.157450 5286 raft_consensus.cc:2781] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.165243 5286 raft_consensus.cc:686] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:24.164707 5103 raft_consensus.cc:348] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.166168 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 291) took 1065 ms. Trace:
I20240503 02:12:24.167328 5115 rpcz_store.cc:276] 0503 02:12:23.100691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.113922 (+ 13231us) service_pool.cc:225] Handling call
0503 02:12:24.121029 (+1007107us) ts_tablet_manager.cc:1120] Deleting tablet 546afc74e6c54f8aa5f7c684484d610b
0503 02:12:24.121038 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.131552 (+ 10514us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.134871 (+ 3319us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.166106 (+ 31235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998704}
I20240503 02:12:24.169760 5115 tablet_replica.cc:330] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.171906 5115 raft_consensus.cc:2229] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.172607 5115 raft_consensus.cc:2258] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.173738 5115 ts_tablet_manager.cc:1875] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.174834 5103 raft_consensus.cc:374] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.176157 5103 raft_consensus.cc:729] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.176450 5103 consensus_queue.cc:260] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.178050 5103 ts_tablet_manager.cc:1398] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:12:24.178459 5103 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.180581 5103 tablet_bootstrap.cc:654] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.175925 5286 consensus_queue.cc:237] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.185978 4810 raft_consensus.cc:2454] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.186484 5105 ts_tablet_manager.cc:1398] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20240503 02:12:24.186786 5105 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.177701 5331 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.188513 5331 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.189502 5331 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.191200 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:24.192061 4746 leader_election.cc:341] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.190384 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.199750 5159 ts_tablet_manager.cc:1909] T 27e8c98d5ea84179a91fdbea8657d6e6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.199893 4671 raft_consensus.cc:2454] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:24.200647 4746 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.200647 5489 tablet.cc:1722] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:24.201403 5331 raft_consensus.cc:2781] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.201516 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 249) took 2259 ms. Trace:
I20240503 02:12:24.201581 5331 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.201781 5331 raft_consensus.cc:3037] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.205264 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b94a5914ab6549d1a420331f4fa47659" } new_table_name: "default.Table_95c5dbfc7e2542dd93ea5839f66200e1" modify_external_catalogs: true
W20240503 02:12:24.193383 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 190176us
I20240503 02:12:24.216488 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c7c7ae25ece740be989d62942a8ae045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.218232 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7c7ae25ece740be989d62942a8ae045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.217276 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e26f10cba82342b0a8719a897d1af394"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:24.218550 5492 tablet.cc:1722] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 02:12:24.218811 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e26f10cba82342b0a8719a897d1af394 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:24.221822 4746 consensus_peers.cc:485] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:24.225786 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e6a3ca705e8482a8cd6a6fb17d93470 consensus state has no leader
I20240503 02:12:24.225920 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet ced251fecca84866ab3ff15bd225e2b7 (table default.Table_4227e90304ee491ca298dabd17aa7b44 [id=536a3d22f6e34248b7936ba2e7a05fd5]) successfully deleted
I20240503 02:12:24.226261 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 14fd4fb88436498999b5f339052ac0b6 (table default.Table_eee1175b622840fe9e2ffb544bcfd426 [id=b6f397e3032e4fe6b1a62c75960ffdaf]) successfully deleted
W20240503 02:12:24.226635 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 53b149e66a0446eeade190dd7f5d2bec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b149e66a0446eeade190dd7f5d2bec already in progress: creating tablet
I20240503 02:12:24.226958 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 9f57f3833a9240c88e0de1b486fae265 (table default.Table_0ccc60c229e74e9197d025a8c631a76c [id=cce029f6965f40e1acb62b9cc9b43102]) successfully deleted
I20240503 02:12:24.227672 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c823155b774e42a5a422994903eae167" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.227916 4810 raft_consensus.cc:2454] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.229234 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 546afc74e6c54f8aa5f7c684484d610b (table default.Table_b8cf4144fb1d4392856c92576ed4dcb1 [id=5ac1fdfc0e4a481486e3120806e473e5]) successfully deleted
W20240503 02:12:24.229681 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 159ea1eb32394b04b72258d24196202b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea1eb32394b04b72258d24196202b already in progress: creating tablet
W20240503 02:12:24.230044 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42e80cc31f5c4908b82f2b030255f6af consensus state has no leader
W20240503 02:12:24.230221 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 66ca08877cd948e1b04736b54bc779db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66ca08877cd948e1b04736b54bc779db already in progress: creating tablet
W20240503 02:12:24.230556 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9327a9e5472b4eda854320c3a95f8e56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9327a9e5472b4eda854320c3a95f8e56 already in progress: creating tablet
W20240503 02:12:24.231011 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c0c18cecefab4b6caed0039d09cc1890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0c18cecefab4b6caed0039d09cc1890 already in progress: creating tablet
I20240503 02:12:24.233928 5106 tablet_bootstrap.cc:492] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.234020 5103 tablet_bootstrap.cc:492] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.234172 5103 ts_tablet_manager.cc:1367] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.004s
I20240503 02:12:24.234162 5106 ts_tablet_manager.cc:1367] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.115s user 0.022s sys 0.011s
I20240503 02:12:24.235054 5103 raft_consensus.cc:348] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.235064 5106 raft_consensus.cc:348] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.235291 5103 raft_consensus.cc:374] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.235327 5106 raft_consensus.cc:374] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.235368 5103 raft_consensus.cc:729] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.235419 5106 raft_consensus.cc:729] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.235670 5106 consensus_queue.cc:260] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.236276 5106 ts_tablet_manager.cc:1398] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:24.236650 5103 consensus_queue.cc:260] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.237469 5106 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.237287 4266 catalog_manager.cc:5484] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.237767 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:24.238876 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]). Expected version 2 got 0
I20240503 02:12:24.239133 5103 ts_tablet_manager.cc:1398] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:24.240388 5103 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.240661 5105 tablet_bootstrap.cc:654] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.242745 5106 tablet_bootstrap.cc:654] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.201633 5159 rpcz_store.cc:276] 0503 02:12:21.941877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.941930 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:24.038729 (+2096799us) ts_tablet_manager.cc:1120] Deleting tablet 27e8c98d5ea84179a91fdbea8657d6e6
0503 02:12:24.098883 (+ 60154us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.104808 (+ 5925us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.120208 (+ 15400us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.201482 (+ 81274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096555}
I20240503 02:12:24.247025 5286 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.247300 5286 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.247519 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 250) took 2297 ms. Trace:
I20240503 02:12:24.247684 5159 rpcz_store.cc:276] 0503 02:12:21.950072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:21.950131 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:24.247349 (+2297218us) ts_tablet_manager.cc:1120] Deleting tablet 95681e67db1e4485876d530133e8a514
0503 02:12:24.247362 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.247479 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296998}
I20240503 02:12:24.248668 5286 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.251726 5115 ts_tablet_manager.cc:1888] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.251770 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 442d795ee01c417dbdeda41fd6853dd7 (DEFAULT_TABLE table=default.Table_4ea17824609f4b25a85121fc6b203812 [id=52915748a54142109318ab8a12dea14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.252300 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d795ee01c417dbdeda41fd6853dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.252543 5103 tablet_bootstrap.cc:654] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.252656 4501 tablet_service.cc:1467] Processing CreateTablet for tablet aa699a1dc3c34be59925560d5e11f49e (DEFAULT_TABLE table=default.Table_cee118c15da24103b1ad3d2b2daf3f52 [id=e3260d7fe9364cc0b19c9caf9a55a94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.253440 5115 log.cc:1196] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/27e8c98d5ea84179a91fdbea8657d6e6
I20240503 02:12:24.253154 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:24.254066 4462 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.254595 5115 ts_tablet_manager.cc:1909] T 27e8c98d5ea84179a91fdbea8657d6e6 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.255271 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce9d746acfe4e9aaea600293470d373 (DEFAULT_TABLE table=default.Table_033e1f689d894383a2e2c0ebbbc6023b [id=009c481538b643a58c9d625b8a184bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.255781 5115 tablet_replica.cc:330] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.255964 5115 raft_consensus.cc:2229] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.256158 5115 raft_consensus.cc:2258] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.256721 5115 ts_tablet_manager.cc:1875] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.257046 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa699a1dc3c34be59925560d5e11f49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.258095 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce9d746acfe4e9aaea600293470d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.259042 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:24.259366 4608 leader_election.cc:341] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d989197a5d6e42379df6d8add449a913
I20240503 02:12:24.259606 4608 leader_election.cc:302] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.259891 5478 raft_consensus.cc:2781] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.260008 5478 raft_consensus.cc:482] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.260093 5478 raft_consensus.cc:3037] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:24.261044 4608 leader_election.cc:341] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: d989197a5d6e42379df6d8add449a913
I20240503 02:12:24.261068 4504 tablet_service.cc:1467] Processing CreateTablet for tablet b8287b45fa0b492ca82c22164f3a7837 (DEFAULT_TABLE table=default.Table_ef5ce757a4e349cab1cdea6c1bc22736 [id=7b088fcf3c334af29998d31075e5367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.261185 4608 leader_election.cc:302] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.261446 5382 raft_consensus.cc:2726] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:24.261519 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8287b45fa0b492ca82c22164f3a7837. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:24.261989 4608 leader_election.cc:341] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: c823155b774e42a5a422994903eae167
I20240503 02:12:24.263074 5478 raft_consensus.cc:504] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.263720 5478 leader_election.cc:288] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.263984 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 442d795ee01c417dbdeda41fd6853dd7 (DEFAULT_TABLE table=default.Table_4ea17824609f4b25a85121fc6b203812 [id=52915748a54142109318ab8a12dea14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.264587 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d795ee01c417dbdeda41fd6853dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.265523 5286 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.265676 5286 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.266626 5286 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.267833 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.268169 4673 raft_consensus.cc:2454] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.268831 4462 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.269367 5286 raft_consensus.cc:2781] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.269778 5286 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.269945 5286 raft_consensus.cc:3037] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.251343 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.271629 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.271858 4810 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.252211 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 442d795ee01c417dbdeda41fd6853dd7 (DEFAULT_TABLE table=default.Table_4ea17824609f4b25a85121fc6b203812 [id=52915748a54142109318ab8a12dea14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.272516 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442d795ee01c417dbdeda41fd6853dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.273200 5478 raft_consensus.cc:482] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.273428 5478 raft_consensus.cc:504] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.274356 5478 leader_election.cc:288] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.254298 4792 tablet_service.cc:1467] Processing CreateTablet for tablet aa699a1dc3c34be59925560d5e11f49e (DEFAULT_TABLE table=default.Table_cee118c15da24103b1ad3d2b2daf3f52 [id=e3260d7fe9364cc0b19c9caf9a55a94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.256711 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce9d746acfe4e9aaea600293470d373 (DEFAULT_TABLE table=default.Table_033e1f689d894383a2e2c0ebbbc6023b [id=009c481538b643a58c9d625b8a184bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.274880 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce9d746acfe4e9aaea600293470d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.275923 5286 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.277523 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:24.277858 4673 raft_consensus.cc:3037] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.278079 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa699a1dc3c34be59925560d5e11f49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.280200 5159 tablet_replica.cc:330] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.280486 5159 raft_consensus.cc:2229] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.280707 5159 raft_consensus.cc:2258] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.281549 5159 ts_tablet_manager.cc:1875] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.281986 4673 raft_consensus.cc:2454] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:24.282509 4462 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.282444 4501 tablet_service.cc:1467] Processing CreateTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 (DEFAULT_TABLE table=default.Table_799bd817690849dc9654a7fb51482f2b [id=abda9019ae0142b78b5e5229f5149509]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.282856 5453 raft_consensus.cc:2781] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.282856 4786 tablet_service.cc:1467] Processing CreateTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 (DEFAULT_TABLE table=default.Table_799bd817690849dc9654a7fb51482f2b [id=abda9019ae0142b78b5e5229f5149509]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.282972 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d5eac44264899a7a372b9f85fb8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.283011 5453 raft_consensus.cc:686] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:24.283252 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d5eac44264899a7a372b9f85fb8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.283278 4509 tablet_service.cc:1467] Processing CreateTablet for tablet b8474c841cb64373a223ad76b43ecf5f (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.283401 5453 consensus_queue.cc:237] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.283725 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8474c841cb64373a223ad76b43ecf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.289057 5286 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.290308 4462 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:24.291288 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.292212 5286 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.292403 5286 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.292604 5286 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.295492 5286 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.296612 5115 ts_tablet_manager.cc:1888] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.297013 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:24.297084 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 27e8c98d5ea84179a91fdbea8657d6e6 (table default.Table_a7538f13504946b6af3145d6c6282670 [id=19642708612b48da91d318e497ba7580]) successfully deleted
I20240503 02:12:24.297062 5115 log.cc:1196] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/894b753b71a04193905a190da320a260
W20240503 02:12:24.297664 4462 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.297891 5115 ts_tablet_manager.cc:1909] T 894b753b71a04193905a190da320a260 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.297945 4501 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 (DEFAULT_TABLE table=default.Table_f1c97fd1492d442e955a652306d0e6a6 [id=68bb420b28734c66bde996fa718da52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.298554 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce4b33f90c40c48cf0a538a1a7fde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.299451 5115 tablet_replica.cc:330] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.300966 5115 raft_consensus.cc:2229] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.302340 5106 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.302575 5106 ts_tablet_manager.cc:1367] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.006s
I20240503 02:12:24.303550 5106 raft_consensus.cc:348] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.303869 5106 raft_consensus.cc:374] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.304332 5106 raft_consensus.cc:729] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:24.304675 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 95681e67db1e4485876d530133e8a514 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95681e67db1e4485876d530133e8a514 already in progress: creating tablet
I20240503 02:12:24.304621 5106 consensus_queue.cc:260] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:24.305150 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b38d63d2cc8c4f54ba43cdf8cde1841f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38d63d2cc8c4f54ba43cdf8cde1841f already in progress: creating tablet
W20240503 02:12:24.305377 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3b41dce88919456aa846dbdcf5e2759f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b41dce88919456aa846dbdcf5e2759f already in progress: creating tablet
W20240503 02:12:24.305591 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ced4dfb2cf541af98d50d22d1a0f0ac already in progress: creating tablet
W20240503 02:12:24.305967 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2f862891b43a4845b723bfdb9a7da591 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f862891b43a4845b723bfdb9a7da591 already in progress: creating tablet
W20240503 02:12:24.306419 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b161628065704c798020edbe098844c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b161628065704c798020edbe098844c2 already in progress: creating tablet
W20240503 02:12:24.306617 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 41d44fe8f6334572996665bdec60e2e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41d44fe8f6334572996665bdec60e2e5 already in progress: creating tablet
I20240503 02:12:24.306763 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 27e8c98d5ea84179a91fdbea8657d6e6 (table default.Table_a7538f13504946b6af3145d6c6282670 [id=19642708612b48da91d318e497ba7580]) successfully deleted
I20240503 02:12:24.301618 5115 raft_consensus.cc:2258] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.308626 5105 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.308975 5105 ts_tablet_manager.cc:1367] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.003s
I20240503 02:12:24.309703 5115 ts_tablet_manager.cc:1875] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.309850 5105 raft_consensus.cc:348] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.310226 5105 raft_consensus.cc:374] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.310333 5105 raft_consensus.cc:729] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.310667 5105 consensus_queue.cc:260] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:24.313232 4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 81651.7us
I20240503 02:12:24.314380 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7ea27632721644359e27e0917381fb1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.315102 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ea27632721644359e27e0917381fb1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.316778 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_681b4fa6666c4e9aaddf611003dcaa44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.317394 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_681b4fa6666c4e9aaddf611003dcaa44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.271494 5331 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.319022 5286 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.319700 5331 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:24.321079 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b84856917cb84c2a853c9a8a80246704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84856917cb84c2a853c9a8a80246704 already in progress: creating tablet
I20240503 02:12:24.321287 4813 raft_consensus.cc:2379] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:24.321928 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.322165 4813 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:24.322749 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79b8db679ba24ac6a93be591c0513ab0 consensus state has no leader
I20240503 02:12:24.325271 5106 ts_tablet_manager.cc:1398] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240503 02:12:24.325441 4813 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:24.325420 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11c306232c4e49ab02be0f195df935 (DEFAULT_TABLE table=default.Table_99049a1e00e84790bce035787060bd61 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.325531 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11c306232c4e49ab02be0f195df935 (DEFAULT_TABLE table=default.Table_99049a1e00e84790bce035787060bd61 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.325925 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11c306232c4e49ab02be0f195df935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.325956 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11c306232c4e49ab02be0f195df935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.326400 5106 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.328807 5106 tablet_bootstrap.cc:654] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.336696 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4702ebe6617c4aebb1264298076c6a4d" } modify_external_catalogs: true
I20240503 02:12:24.336944 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4702ebe6617c4aebb1264298076c6a4d" } modify_external_catalogs: true
I20240503 02:12:24.338069 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:24.338814 5478 raft_consensus.cc:482] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.338924 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:24.339277 4673 raft_consensus.cc:2379] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:24.339107 5478 raft_consensus.cc:504] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.340008 5478 leader_election.cc:288] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.340512 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6616d554610d429190adcee205cf086c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.341396 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6616d554610d429190adcee205cf086c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.345989 4462 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:24.347203 4746 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.348701 4652 tablet_service.cc:1467] Processing CreateTablet for tablet aa699a1dc3c34be59925560d5e11f49e (DEFAULT_TABLE table=default.Table_cee118c15da24103b1ad3d2b2daf3f52 [id=e3260d7fe9364cc0b19c9caf9a55a94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.349344 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa699a1dc3c34be59925560d5e11f49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.354612 5286 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.354826 5286 raft_consensus.cc:686] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:24.355118 5286 consensus_queue.cc:237] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.356462 5103 tablet_bootstrap.cc:492] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.356721 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c (DEFAULT_TABLE table=default.Table_8af9d8455d574ede92363cd1cafb1334 [id=f7f88566af964cefb2026fb433ca0f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.357431 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b8474c841cb64373a223ad76b43ecf5f (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.357354 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11c306232c4e49ab02be0f195df935 (DEFAULT_TABLE table=default.Table_99049a1e00e84790bce035787060bd61 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.357854 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11c306232c4e49ab02be0f195df935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.357898 4637 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 (DEFAULT_TABLE table=default.Table_f1c97fd1492d442e955a652306d0e6a6 [id=68bb420b28734c66bde996fa718da52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.357647 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd296bb3b574d6cbb0252ad10ada00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.358314 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce4b33f90c40c48cf0a538a1a7fde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.353375 4786 tablet_service.cc:1467] Processing CreateTablet for tablet b8474c841cb64373a223ad76b43ecf5f (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.357730 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c823155b774e42a5a422994903eae167" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.358841 4813 raft_consensus.cc:3037] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.359045 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8474c841cb64373a223ad76b43ecf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.358558 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c (DEFAULT_TABLE table=default.Table_8af9d8455d574ede92363cd1cafb1334 [id=f7f88566af964cefb2026fb433ca0f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.360571 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd296bb3b574d6cbb0252ad10ada00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.361311 4643 tablet_service.cc:1467] Processing CreateTablet for tablet b8287b45fa0b492ca82c22164f3a7837 (DEFAULT_TABLE table=default.Table_ef5ce757a4e349cab1cdea6c1bc22736 [id=7b088fcf3c334af29998d31075e5367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.361704 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8287b45fa0b492ca82c22164f3a7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.362236 4813 raft_consensus.cc:2454] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.364835 5105 ts_tablet_manager.cc:1398] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20240503 02:12:24.365221 5105 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.365243 4649 tablet_service.cc:1467] Processing CreateTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 (DEFAULT_TABLE table=default.Table_799bd817690849dc9654a7fb51482f2b [id=abda9019ae0142b78b5e5229f5149509]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.365600 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d5eac44264899a7a372b9f85fb8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.366966 5105 tablet_bootstrap.cc:654] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.369604 4792 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 (DEFAULT_TABLE table=default.Table_f1c97fd1492d442e955a652306d0e6a6 [id=68bb420b28734c66bde996fa718da52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.369994 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce4b33f90c40c48cf0a538a1a7fde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.370194 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce9d746acfe4e9aaea600293470d373 (DEFAULT_TABLE table=default.Table_033e1f689d894383a2e2c0ebbbc6023b [id=009c481538b643a58c9d625b8a184bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.370513 4781 tablet_service.cc:1467] Processing CreateTablet for tablet b8287b45fa0b492ca82c22164f3a7837 (DEFAULT_TABLE table=default.Table_ef5ce757a4e349cab1cdea6c1bc22736 [id=7b088fcf3c334af29998d31075e5367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.370606 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce9d746acfe4e9aaea600293470d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.370855 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8287b45fa0b492ca82c22164f3a7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.371158 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.356722 5103 ts_tablet_manager.cc:1367] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.117s user 0.020s sys 0.042s
I20240503 02:12:24.371423 4813 raft_consensus.cc:2454] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.372274 5103 raft_consensus.cc:348] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.372501 5103 raft_consensus.cc:374] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.372591 5103 raft_consensus.cc:729] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.372807 5103 consensus_queue.cc:260] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.373049 5453 raft_consensus.cc:482] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.373193 5453 raft_consensus.cc:504] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:24.373526 5103 ts_tablet_manager.cc:1398] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:24.373842 5453 leader_election.cc:288] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.373818 5103 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.374343 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8474c841cb64373a223ad76b43ecf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.375855 5103 tablet_bootstrap.cc:654] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.375880 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a596cfe654aa99e83d070b7884e6b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.377279 5426 raft_consensus.cc:2726] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:24.378516 5478 raft_consensus.cc:482] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.378664 5478 raft_consensus.cc:504] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.379293 5478 leader_election.cc:288] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.385367 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 894b753b71a04193905a190da320a260 (table default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]) successfully deleted
I20240503 02:12:24.388584 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.389026 4813 raft_consensus.cc:2454] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.393601 4608 leader_election.cc:302] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.396627 5513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:43321 every 8 connection(s)
I20240503 02:12:24.397579 5268 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:43321
I20240503 02:12:24.399775 5268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:12:24.401024 5478 raft_consensus.cc:2781] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.401074 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 5268
I20240503 02:12:24.401194 5478 raft_consensus.cc:686] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:24.401491 5478 consensus_queue.cc:237] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.410001 4608 leader_election.cc:302] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.412508 5159 ts_tablet_manager.cc:1888] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.413512 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:24.414655 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:24.416402 4462 leader_election.cc:341] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 894b753b71a04193905a190da320a260
I20240503 02:12:24.418494 5382 raft_consensus.cc:2781] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.418679 5382 raft_consensus.cc:482] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.418788 5382 raft_consensus.cc:3037] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.422864 5382 raft_consensus.cc:504] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.424041 5382 leader_election.cc:288] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.425772 4608 leader_election.cc:302] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.428421 5382 raft_consensus.cc:2781] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.428992 5382 raft_consensus.cc:482] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.429244 5382 raft_consensus.cc:3037] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.433780 5382 raft_consensus.cc:504] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.434680 5382 leader_election.cc:288] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.441834 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c (DEFAULT_TABLE table=default.Table_8af9d8455d574ede92363cd1cafb1334 [id=f7f88566af964cefb2026fb433ca0f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.442307 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd296bb3b574d6cbb0252ad10ada00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.442909 5103 tablet_bootstrap.cc:492] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.443159 5103 ts_tablet_manager.cc:1367] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.069s user 0.033s sys 0.035s
I20240503 02:12:24.444120 5103 raft_consensus.cc:348] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.444434 5103 raft_consensus.cc:374] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.444551 5103 raft_consensus.cc:729] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.444864 5103 consensus_queue.cc:260] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.445480 5103 ts_tablet_manager.cc:1398] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:24.445890 5103 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:24.447172 4608 leader_election.cc:341] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: c823155b774e42a5a422994903eae167
W20240503 02:12:24.447566 4608 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.447640 5103 tablet_bootstrap.cc:654] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:24.447957 4608 leader_election.cc:341] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 387a596cfe654aa99e83d070b7884e6b
I20240503 02:12:24.448091 4608 leader_election.cc:302] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.448812 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:24.449981 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:24.412986 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894b753b71a04193905a190da320a260" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.426018 5331 raft_consensus.cc:482] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.427407 5105 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.442261 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1210a1217247b9801226055e2f2195" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.447125 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd402c0eb9e4cf49dc722f2dd214e09" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.454941 4813 raft_consensus.cc:2454] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.454941 5331 raft_consensus.cc:504] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.455351 4810 raft_consensus.cc:3037] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.456012 5331 leader_election.cc:288] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.456049 5103 tablet_bootstrap.cc:492] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.456185 5103 ts_tablet_manager.cc:1367] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:12:24.455377 5105 ts_tablet_manager.cc:1367] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20240503 02:12:24.457115 5103 raft_consensus.cc:348] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.457196 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a51563396614c54b980f0679e84a2b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.457316 5103 raft_consensus.cc:374] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.457398 5103 raft_consensus.cc:729] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.455404 4811 raft_consensus.cc:3037] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.457623 5103 consensus_queue.cc:260] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.457932 5105 raft_consensus.cc:348] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.458142 5105 raft_consensus.cc:374] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.458189 5103 ts_tablet_manager.cc:1398] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:24.458241 5105 raft_consensus.cc:729] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.458489 5103 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.458642 4810 raft_consensus.cc:2454] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.459836 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a51563396614c54b980f0679e84a2b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:24.460062 4528 raft_consensus.cc:2454] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:24.460183 5103 tablet_bootstrap.cc:654] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.460496 4811 raft_consensus.cc:2454] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.460608 4746 leader_election.cc:302] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.461074 4462 leader_election.cc:302] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:24.461077 4746 leader_election.cc:341] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.461475 5331 raft_consensus.cc:2781] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.461488 5286 raft_consensus.cc:2781] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.461578 5286 raft_consensus.cc:482] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.461633 5331 raft_consensus.cc:482] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.461676 5286 raft_consensus.cc:3037] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.461719 5331 raft_consensus.cc:3037] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.464324 5286 raft_consensus.cc:504] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:24.464978 5286 leader_election.cc:288] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.466128 5331 raft_consensus.cc:504] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.466383 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894b753b71a04193905a190da320a260" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.466693 4811 raft_consensus.cc:3037] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.467214 5331 leader_election.cc:288] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.468043 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a51563396614c54b980f0679e84a2b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:24.468319 4528 raft_consensus.cc:3037] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.470278 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a51563396614c54b980f0679e84a2b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:24.470907 4746 leader_election.cc:341] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.471299 4528 raft_consensus.cc:2454] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:24.455230 5159 log.cc:1196] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/24737f8a8ca74996b9f54d49d12b9288
I20240503 02:12:24.471704 4746 leader_election.cc:302] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:24.472200 4462 leader_election.cc:341] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 894b753b71a04193905a190da320a260
I20240503 02:12:24.472290 5159 ts_tablet_manager.cc:1909] T 24737f8a8ca74996b9f54d49d12b9288 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.471792 4811 raft_consensus.cc:2454] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:24.472419 5331 raft_consensus.cc:2781] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.472842 5103 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.473086 5103 ts_tablet_manager.cc:1367] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 02:12:24.473517 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 251) took 2338 ms. Trace:
I20240503 02:12:24.472700 5331 raft_consensus.cc:686] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:24.473672 5159 rpcz_store.cc:276] 0503 02:12:22.134736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.134793 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:24.280126 (+2145333us) ts_tablet_manager.cc:1120] Deleting tablet 24737f8a8ca74996b9f54d49d12b9288
0503 02:12:24.280136 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.314348 (+ 34212us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.412325 (+ 97977us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.473451 (+ 61126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2145073}
I20240503 02:12:24.473918 5103 raft_consensus.cc:348] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.474229 5103 raft_consensus.cc:374] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.474138 5331 consensus_queue.cc:237] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.474306 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 24737f8a8ca74996b9f54d49d12b9288 (table default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]) successfully deleted
I20240503 02:12:24.474359 5103 raft_consensus.cc:729] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.474601 5103 consensus_queue.cc:260] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.474897 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 252) took 2310 ms. Trace:
I20240503 02:12:24.475071 5159 rpcz_store.cc:276] 0503 02:12:22.164000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.164058 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:24.474698 (+2310640us) ts_tablet_manager.cc:1120] Deleting tablet d022b5df4a914e439fff918f70c34157
0503 02:12:24.474708 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.474842 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310419}
I20240503 02:12:24.475322 5103 ts_tablet_manager.cc:1398] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:24.475416 4462 leader_election.cc:302] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:24.475699 5103 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.475761 5286 raft_consensus.cc:2781] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.475880 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 253) took 2309 ms. Trace:
I20240503 02:12:24.475927 5286 raft_consensus.cc:686] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:24.476047 5159 rpcz_store.cc:276] 0503 02:12:22.166234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.166287 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:24.475720 (+2309433us) ts_tablet_manager.cc:1120] Deleting tablet b91abaf51abb4c2ca65412c3771c1ef2
0503 02:12:24.475730 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.475830 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309180}
I20240503 02:12:24.476218 5286 consensus_queue.cc:237] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:24.476606 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b91abaf51abb4c2ca65412c3771c1ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b91abaf51abb4c2ca65412c3771c1ef2 already in progress: creating tablet
I20240503 02:12:24.477062 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 254) took 2293 ms. Trace:
I20240503 02:12:24.477587 5103 tablet_bootstrap.cc:654] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.478099 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 27c13f257eae44689af07cc31ea254c4
I20240503 02:12:24.479372 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.479728 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3d4325dad27b49bfa8a8e38c55372f6c" } new_table_name: "default.Table_c5ef79d34f424f519e36c63857a7cce9" modify_external_catalogs: true
W20240503 02:12:24.480175 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 27c13f257eae44689af07cc31ea254c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27c13f257eae44689af07cc31ea254c4 already in progress: creating tablet
I20240503 02:12:24.482463 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 34eb69d5f4bd427a8e430a00fb076f83
I20240503 02:12:24.483244 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 2152f205cf1b4ec7ae03605bbfa007e1 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.484242 5478 raft_consensus.cc:2726] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:24.484987 4608 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:24.485546 4608 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:24.485826 4608 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.486232 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:24.487001 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 34eb69d5f4bd427a8e430a00fb076f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34eb69d5f4bd427a8e430a00fb076f83 already in progress: creating tablet
I20240503 02:12:24.487440 4608 leader_election.cc:302] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.487751 5478 raft_consensus.cc:2781] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.488009 5478 raft_consensus.cc:686] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
W20240503 02:12:24.488407 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 84b6e303e34a465caffed6c9e1e6bf74 (table default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 84b6e303e34a465caffed6c9e1e6bf74 consensus state has no leader
I20240503 02:12:24.488498 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]). Expected version 2 got 0
I20240503 02:12:24.488446 5478 consensus_queue.cc:237] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.488960 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 22eec2c9a5654a4a8d6c3011ebc3c67c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.489377 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.489658 5114 tablet_replica.cc:330] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:24.489917 5114 raft_consensus.cc:2229] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.490238 5114 pending_rounds.cc:62] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:24.490269 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet e94031465ce7407dab0f90ad8809ac21 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.490325 5114 pending_rounds.cc:68] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420801772777472 op_type: NO_OP noop_request { }
I20240503 02:12:24.490478 5114 raft_consensus.cc:2866] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:24.490608 5114 raft_consensus.cc:2258] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.491907 5103 tablet_bootstrap.cc:492] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.492127 5103 ts_tablet_manager.cc:1367] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240503 02:12:24.492283 5114 ts_tablet_manager.cc:1875] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.493192 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 3e6a3ca705e8482a8cd6a6fb17d93470 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.493507 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.494818 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet aa67857b858c4719b498291c52b7aceb
I20240503 02:12:24.495425 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ed525a61c734ee2b23d5d4f319a7d67 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.495796 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67857b858c4719b498291c52b7aceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.479429 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.458521 5105 consensus_queue.cc:260] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.498840 5105 ts_tablet_manager.cc:1398] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20240503 02:12:24.499321 5105 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.499339 5114 ts_tablet_manager.cc:1888] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.500550 5114 log.cc:1196] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/24737f8a8ca74996b9f54d49d12b9288
I20240503 02:12:24.501219 5114 ts_tablet_manager.cc:1909] T 24737f8a8ca74996b9f54d49d12b9288 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:24.502393 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 24737f8a8ca74996b9f54d49d12b9288 (table default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]) successfully deleted
I20240503 02:12:24.500304 5103 raft_consensus.cc:348] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.502744 5103 raft_consensus.cc:374] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.502888 5103 raft_consensus.cc:729] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.503191 5103 consensus_queue.cc:260] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.503865 5103 ts_tablet_manager.cc:1398] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:24.504452 5103 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.505111 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67857b858c4719b498291c52b7aceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.486266 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.503037 5159 rpcz_store.cc:276] 0503 02:12:22.183901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.183957 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:24.476897 (+2292940us) ts_tablet_manager.cc:1120] Deleting tablet 959d998ec4ec4a52a6031650f2da4691
0503 02:12:24.476906 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.477012 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292723}
W20240503 02:12:24.506680 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 959d998ec4ec4a52a6031650f2da4691 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959d998ec4ec4a52a6031650f2da4691 already in progress: creating tablet
W20240503 02:12:24.506924 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet aa67857b858c4719b498291c52b7aceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67857b858c4719b498291c52b7aceb already in progress: creating tablet
I20240503 02:12:24.507652 4608 leader_election.cc:302] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.507835 5103 tablet_bootstrap.cc:654] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.508042 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 255) took 2262 ms. Trace:
I20240503 02:12:24.508708 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.513048 5159 rpcz_store.cc:276] 0503 02:12:22.245120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.245190 (+ 70us) service_pool.cc:225] Handling call
0503 02:12:24.506653 (+2261463us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:24.506667 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.507988 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2261185}
W20240503 02:12:24.514117 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22776c81d1a24d368abb68274cf32a0b already in progress: creating tablet
I20240503 02:12:24.514674 5382 raft_consensus.cc:2781] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.514922 5382 raft_consensus.cc:686] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:24.515316 5382 consensus_queue.cc:237] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.524333 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.509860 5105 tablet_bootstrap.cc:654] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.526877 4575 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:24.527731 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67857b858c4719b498291c52b7aceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.528105 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.540056 5159 tablet_replica.cc:330] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.540427 5159 raft_consensus.cc:2229] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.541132 5159 raft_consensus.cc:2258] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.541913 5159 ts_tablet_manager.cc:1875] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.544548 4859 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:24.547407 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a9cfba004dfe4c7e8def7c54b105546f" } modify_external_catalogs: true
I20240503 02:12:24.547698 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a9cfba004dfe4c7e8def7c54b105546f" } modify_external_catalogs: true
I20240503 02:12:24.565656 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 87713e899d2148308316d807a6d76921
I20240503 02:12:24.561102 5514 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:12:24.567970 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.569132 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.570595 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:24.573100 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 87713e899d2148308316d807a6d76921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87713e899d2148308316d807a6d76921 already in progress: creating tablet
I20240503 02:12:24.575681 5106 tablet_bootstrap.cc:492] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.575984 5106 ts_tablet_manager.cc:1367] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.250s user 0.005s sys 0.001s
I20240503 02:12:24.577275 5106 raft_consensus.cc:348] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.578073 5106 raft_consensus.cc:374] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.578223 5106 raft_consensus.cc:729] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.578678 5106 consensus_queue.cc:260] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.579465 5106 ts_tablet_manager.cc:1398] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:24.579964 5106 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.582573 5514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap starting.
I20240503 02:12:24.582902 5106 tablet_bootstrap.cc:654] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.618495 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_deade72f82eb4fd18e15b3360d826a9a" } new_table_name: "default.Table_8ac2cdb3ebec4a2795b7fa6e47dc56b8" modify_external_catalogs: true
W20240503 02:12:24.619984 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b14241dd76864e8a93d0279d3ab8f3ac consensus state has no leader
I20240503 02:12:24.621977 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:24.622149 5105 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.622342 5105 ts_tablet_manager.cc:1367] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.005s
I20240503 02:12:24.622697 4268 catalog_manager.cc:6554] ReplaceTablet: tablet e94031465ce7407dab0f90ad8809ac21 deleted and replaced by tablet a88710ac9062472f80bf6fcfb82793a0
I20240503 02:12:24.623358 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.624033 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d47e1a80afa040948b01f4ba1d3bf10b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.624673 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet e94031465ce7407dab0f90ad8809ac21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94031465ce7407dab0f90ad8809ac21 already in progress: creating tablet
W20240503 02:12:24.624897 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d47e1a80afa040948b01f4ba1d3bf10b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.626421 5105 raft_consensus.cc:348] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.626705 5105 raft_consensus.cc:374] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.626802 5105 raft_consensus.cc:729] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.627120 5105 consensus_queue.cc:260] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.625598 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1b6529a1250745eea5d10a3f304a28e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:24.627876 5105 ts_tablet_manager.cc:1398] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
W20240503 02:12:24.627948 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b6529a1250745eea5d10a3f304a28e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.605438 4721 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:24.607318 5115 ts_tablet_manager.cc:1888] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.633739 5115 log.cc:1196] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7d5e627d818245e0bdeb16531499f7d5
I20240503 02:12:24.634435 5115 ts_tablet_manager.cc:1909] T 7d5e627d818245e0bdeb16531499f7d5 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.636970 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.640053 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7d5e627d818245e0bdeb16531499f7d5 (table default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]) successfully deleted
I20240503 02:12:24.640146 5115 tablet_replica.cc:330] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.640401 5115 raft_consensus.cc:2229] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.640753 5115 pending_rounds.cc:62] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:24.641197 5159 ts_tablet_manager.cc:1888] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.641381 5159 log.cc:1196] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/bf9c07c7cbb743d4a8e9610281a00f94
I20240503 02:12:24.641861 5105 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.642258 5159 ts_tablet_manager.cc:1909] T bf9c07c7cbb743d4a8e9610281a00f94 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.643050 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 256) took 2212 ms. Trace:
I20240503 02:12:24.643374 5159 rpcz_store.cc:276] 0503 02:12:22.430497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.430565 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:24.523459 (+2092894us) ts_tablet_manager.cc:1120] Deleting tablet bf9c07c7cbb743d4a8e9610281a00f94
0503 02:12:24.539698 (+ 16239us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.557960 (+ 18262us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.636282 (+ 78322us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.643007 (+ 6725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092659}
I20240503 02:12:24.644081 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet bf9c07c7cbb743d4a8e9610281a00f94 (table default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]) successfully deleted
I20240503 02:12:24.645725 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9ed525a61c734ee2b23d5d4f319a7d67
I20240503 02:12:24.646739 4272 catalog_manager.cc:6554] ReplaceTablet: tablet 9ed525a61c734ee2b23d5d4f319a7d67 deleted and replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544
I20240503 02:12:24.648342 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.649590 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.649621 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.649935 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:24.650665 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9ed525a61c734ee2b23d5d4f319a7d67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed525a61c734ee2b23d5d4f319a7d67 already in progress: creating tablet
I20240503 02:12:24.650732 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8cdd736fadb64d9faf9eb743916fa0b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.651566 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cdd736fadb64d9faf9eb743916fa0b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.651634 5105 tablet_bootstrap.cc:654] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.653616 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 266) took 2123 ms. Trace:
I20240503 02:12:24.653796 5159 rpcz_store.cc:276] 0503 02:12:22.528230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.530410 (+ 2180us) service_pool.cc:225] Handling call
0503 02:12:24.643809 (+2113399us) ts_tablet_manager.cc:1120] Deleting tablet 035216c9475d4b59aa1ac81b285b4d8a
0503 02:12:24.643818 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.652126 (+ 8308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2110422}
I20240503 02:12:24.654520 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 269) took 2103 ms. Trace:
W20240503 02:12:24.654803 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 035216c9475d4b59aa1ac81b285b4d8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 035216c9475d4b59aa1ac81b285b4d8a already in progress: creating tablet
I20240503 02:12:24.655110 5159 rpcz_store.cc:276] 0503 02:12:22.551546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.551726 (+ 180us) service_pool.cc:225] Handling call
0503 02:12:24.654361 (+2102635us) ts_tablet_manager.cc:1120] Deleting tablet 20ce11c70a924b8d92d8f36fc7401dfd
0503 02:12:24.654375 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.654486 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101296}
I20240503 02:12:24.655697 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 270) took 2017 ms. Trace:
W20240503 02:12:24.655966 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20ce11c70a924b8d92d8f36fc7401dfd already in progress: creating tablet
I20240503 02:12:24.656280 5159 rpcz_store.cc:276] 0503 02:12:22.638752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.640546 (+ 1794us) service_pool.cc:225] Handling call
0503 02:12:24.655574 (+2015028us) ts_tablet_manager.cc:1120] Deleting tablet 20ce11c70a924b8d92d8f36fc7401dfd
0503 02:12:24.655583 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.655669 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004753}
W20240503 02:12:24.656950 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 20ce11c70a924b8d92d8f36fc7401dfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20ce11c70a924b8d92d8f36fc7401dfd already in progress: creating tablet
I20240503 02:12:24.657389 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 272) took 2005 ms. Trace:
I20240503 02:12:24.657613 5159 rpcz_store.cc:276] 0503 02:12:22.652118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.654238 (+ 2120us) service_pool.cc:225] Handling call
0503 02:12:24.657215 (+2002977us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:24.657226 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.657355 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001903}
W20240503 02:12:24.658067 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 53017352b8f34955b116bbfd32f04854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53017352b8f34955b116bbfd32f04854 already in progress: creating tablet
I20240503 02:12:24.658411 5159 tablet_replica.cc:330] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.659773 5159 raft_consensus.cc:2229] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.660320 5159 raft_consensus.cc:2258] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.661679 5159 ts_tablet_manager.cc:1875] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.669811 5331 raft_consensus.cc:482] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.670042 5331 raft_consensus.cc:504] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.670853 5331 leader_election.cc:288] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.671550 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.671700 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:24.672259 4746 leader_election.cc:341] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:24.672497 4746 leader_election.cc:341] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.672631 4746 leader_election.cc:302] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.673134 5331 raft_consensus.cc:2726] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:24.674731 5103 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.674938 5103 ts_tablet_manager.cc:1367] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.171s user 0.059s sys 0.092s
I20240503 02:12:24.675041 5115 pending_rounds.cc:68] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420803033444352 op_type: NO_OP noop_request { }
I20240503 02:12:24.675657 5115 raft_consensus.cc:2866] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:24.675859 5103 raft_consensus.cc:348] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.676086 5103 raft_consensus.cc:374] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.676163 5103 raft_consensus.cc:729] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.676539 5115 raft_consensus.cc:2258] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.676445 5103 consensus_queue.cc:260] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.680716 5103 ts_tablet_manager.cc:1398] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:24.681106 5103 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.683365 5115 ts_tablet_manager.cc:1875] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.685767 5103 tablet_bootstrap.cc:654] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.684171 5106 tablet_bootstrap.cc:492] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.689224 5106 ts_tablet_manager.cc:1367] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.003s
W20240503 02:12:24.693591 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbfe6b2b9cab4517ae7cabec4796c9ce consensus state has no leader
I20240503 02:12:24.695340 5103 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:24.695499 5103 ts_tablet_manager.cc:1367] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 02:12:24.695544 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d607da9689c14e51add9178a48b35913
I20240503 02:12:24.698943 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet d607da9689c14e51add9178a48b35913 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.699553 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet d607da9689c14e51add9178a48b35913 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.699640 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet d607da9689c14e51add9178a48b35913 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:24.699739 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d607da9689c14e51add9178a48b35913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d607da9689c14e51add9178a48b35913 already in progress: creating tablet
I20240503 02:12:24.701978 5115 ts_tablet_manager.cc:1888] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.702193 5115 log.cc:1196] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7d1210a1217247b9801226055e2f2195
I20240503 02:12:24.702703 5115 ts_tablet_manager.cc:1909] T 7d1210a1217247b9801226055e2f2195 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.703361 5115 tablet_replica.cc:330] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.703433 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7d1210a1217247b9801226055e2f2195 (table default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]) successfully deleted
I20240503 02:12:24.703640 5115 raft_consensus.cc:2229] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.703907 5115 raft_consensus.cc:2258] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.704547 5115 ts_tablet_manager.cc:1875] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.705024 5103 raft_consensus.cc:348] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.705271 5103 raft_consensus.cc:374] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.705387 5103 raft_consensus.cc:729] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.705533 5286 consensus_queue.cc:1035] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.705669 5103 consensus_queue.cc:260] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.708580 5105 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:24.711027 5105 ts_tablet_manager.cc:1367] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.006s
I20240503 02:12:24.712602 5105 raft_consensus.cc:348] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.713058 5105 raft_consensus.cc:374] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.713270 5105 raft_consensus.cc:729] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.713943 5159 ts_tablet_manager.cc:1888] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.713783 5105 consensus_queue.cc:260] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.714095 5103 ts_tablet_manager.cc:1398] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 02:12:24.715060 5105 ts_tablet_manager.cc:1398] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:24.715730 5105 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:24.720305 5105 tablet_bootstrap.cc:654] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.717996 5106 raft_consensus.cc:348] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.720834 5106 raft_consensus.cc:374] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.720968 5106 raft_consensus.cc:729] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.721932 5106 consensus_queue.cc:260] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.722548 5106 ts_tablet_manager.cc:1398] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20240503 02:12:24.724627 5103 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:24.726884 5103 tablet_bootstrap.cc:654] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.730249 5106 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.731285 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_86bc1fe3319346129d38c1c2d731742f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.732100 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86bc1fe3319346129d38c1c2d731742f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.733563 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6a4f81fe56f84e4ab87ea63d67015ae2" } new_table_name: "default.Table_55c37198f9d2480784f643011b70dee8" modify_external_catalogs: true
I20240503 02:12:24.735926 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7e6b24685afa41e388162e59492f00b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.736454 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e6b24685afa41e388162e59492f00b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.720129 5115 ts_tablet_manager.cc:1888] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.746855 5115 log.cc:1196] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/d989197a5d6e42379df6d8add449a913
I20240503 02:12:24.747442 5159 log.cc:1196] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/86d719c58974442281c7d694e1ca485e
I20240503 02:12:24.748051 5159 ts_tablet_manager.cc:1909] T 86d719c58974442281c7d694e1ca485e P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.748638 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6c6c926459ab48ef88b1da1b7358dbdd" } new_table_name: "default.Table_63229f7088d0478b9de9fa953f84866b" modify_external_catalogs: true
I20240503 02:12:24.748998 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 273) took 2092 ms. Trace:
I20240503 02:12:24.749099 5159 rpcz_store.cc:276] 0503 02:12:22.656039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.656172 (+ 133us) service_pool.cc:225] Handling call
0503 02:12:24.658343 (+2002171us) ts_tablet_manager.cc:1120] Deleting tablet 86d719c58974442281c7d694e1ca485e
0503 02:12:24.658352 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.710456 (+ 52104us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.713816 (+ 3360us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.748939 (+ 35123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1996760}
I20240503 02:12:24.749588 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 86d719c58974442281c7d694e1ca485e (table default.Table_af0acc1c80404f3a875b88f459374f81 [id=835bd7136ace4de192d4c5e1965bff97]) successfully deleted
I20240503 02:12:24.748600 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_033290f38d39444aad4953c5f91ffaeb" } new_table_name: "default.Table_5df26c990f414242ac526c32679385fe" modify_external_catalogs: true
I20240503 02:12:24.752368 5115 ts_tablet_manager.cc:1909] T d989197a5d6e42379df6d8add449a913 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:24.753142 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 287) took 2073 ms. Trace:
I20240503 02:12:24.753322 5159 rpcz_store.cc:276] 0503 02:12:22.680113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.680175 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:24.752946 (+2072771us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:24.752959 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.753084 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2072510}
I20240503 02:12:24.747212 5514 log.cc:824] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Log is configured to *not* fsync() on all Append() calls
W20240503 02:12:24.754540 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8750b02c5936412db525d2d3e77befeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8750b02c5936412db525d2d3e77befeb already in progress: creating tablet
I20240503 02:12:24.754825 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet d989197a5d6e42379df6d8add449a913 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
I20240503 02:12:24.755295 5159 tablet_replica.cc:330] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.755550 5159 raft_consensus.cc:2229] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.755798 5159 raft_consensus.cc:2258] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.756440 5159 ts_tablet_manager.cc:1875] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.751879 5106 tablet_bootstrap.cc:654] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.757066 5331 raft_consensus.cc:482] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.757253 5331 raft_consensus.cc:504] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.757930 5331 leader_election.cc:288] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.758277 5115 tablet_replica.cc:330] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.758977 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.759068 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:24.759735 4746 leader_election.cc:341] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.759953 5115 raft_consensus.cc:2229] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:24.760006 4746 leader_election.cc:341] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:24.760116 4746 leader_election.cc:302] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:24.760434 5115 pending_rounds.cc:62] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:24.760594 5115 pending_rounds.cc:68] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420803139190784 op_type: NO_OP noop_request { }
I20240503 02:12:24.760687 5331 raft_consensus.cc:2726] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:24.760825 5115 raft_consensus.cc:2866] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:24.761034 5115 raft_consensus.cc:2258] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.761993 5115 ts_tablet_manager.cc:1875] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.762527 5455 raft_consensus.cc:482] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.762691 5455 raft_consensus.cc:504] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.763469 5455 leader_election.cc:288] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.764395 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c07c7cbb743d4a8e9610281a00f94" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.764614 4673 raft_consensus.cc:2454] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.765499 4462 leader_election.cc:302] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:24.765887 5455 raft_consensus.cc:2781] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.765990 5455 raft_consensus.cc:482] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.766064 5455 raft_consensus.cc:3037] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.766252 5106 tablet_bootstrap.cc:492] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.766453 5106 ts_tablet_manager.cc:1367] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
I20240503 02:12:24.767483 5106 raft_consensus.cc:348] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.767767 5106 raft_consensus.cc:374] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.767849 5106 raft_consensus.cc:729] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.768194 5106 consensus_queue.cc:260] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.768930 5106 ts_tablet_manager.cc:1398] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:24.769330 5106 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:24.774611 4746 consensus_peers.cc:485] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d5e627d818245e0bdeb16531499f7d5. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:24.775056 4462 leader_election.cc:341] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: bf9c07c7cbb743d4a8e9610281a00f94
I20240503 02:12:24.775266 5106 tablet_bootstrap.cc:654] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.777966 5159 ts_tablet_manager.cc:1888] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.778788 5455 raft_consensus.cc:504] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.789155 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c07c7cbb743d4a8e9610281a00f94" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:24.792989 4462 leader_election.cc:341] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: bf9c07c7cbb743d4a8e9610281a00f94
I20240503 02:12:24.793272 5455 leader_election.cc:288] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.799993 4273 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]). Expected version 2 got 0
I20240503 02:12:24.801869 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1 (DEFAULT_TABLE table=default.Table_c7c7ae25ece740be989d62942a8ae045 [id=1b3b3d86efe94e1d8f6230f7ceb285cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.802326 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.802464 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1 (DEFAULT_TABLE table=default.Table_c7c7ae25ece740be989d62942a8ae045 [id=1b3b3d86efe94e1d8f6230f7ceb285cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.803486 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:24.803835 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]). Expected version 3 got 0
I20240503 02:12:24.804286 4271 catalog_manager.cc:5484] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.804723 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:24.805181 4501 tablet_service.cc:1467] Processing CreateTablet for tablet a86ac17982324847949b8a9245329489 (DEFAULT_TABLE table=default.Table_e26f10cba82342b0a8719a897d1af394 [id=4dbb0ce32dad4373a8362c9297990767]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.805584 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86ac17982324847949b8a9245329489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.806288 4271 catalog_manager.cc:5484] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.806692 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]). Expected version 2 got 0
I20240503 02:12:24.806947 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
I20240503 02:12:24.803562 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1 (DEFAULT_TABLE table=default.Table_c7c7ae25ece740be989d62942a8ae045 [id=1b3b3d86efe94e1d8f6230f7ceb285cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.809654 5159 log.cc:1196] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/894b753b71a04193905a190da320a260
I20240503 02:12:24.809932 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.810127 5159 ts_tablet_manager.cc:1909] T 894b753b71a04193905a190da320a260 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.810703 4673 raft_consensus.cc:3037] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:24.813495 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea4dac17dca3442d8934ac357779b78a consensus state has no leader
I20240503 02:12:24.813553 4673 raft_consensus.cc:2454] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:24.813846 5115 ts_tablet_manager.cc:1888] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.813974 5115 log.cc:1196] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:24.814448 5115 ts_tablet_manager.cc:1909] T 6cd402c0eb9e4cf49dc722f2dd214e09 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:24.814610 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a88710ac9062472f80bf6fcfb82793a0 has no consensus state
I20240503 02:12:24.814913 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 (DEFAULT_TABLE table=default.Table_6e29e77c800340dbad41e6a6b3c1c0ec [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.815425 5115 tablet_replica.cc:330] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:24.815555 4650 tablet_service.cc:1467] Processing CreateTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe (DEFAULT_TABLE table=default.Table_d6e976e60b3e445492636b06856344c3 [id=0f32eabc16cd47bdb6addf705adfffa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.815654 5115 raft_consensus.cc:2229] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.815892 5115 pending_rounds.cc:62] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:24.815976 5115 pending_rounds.cc:68] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420802671566848 op_type: NO_OP noop_request { }
I20240503 02:12:24.816110 5115 raft_consensus.cc:2866] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:24.816231 5115 raft_consensus.cc:2258] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.817862 4789 tablet_service.cc:1467] Processing CreateTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe (DEFAULT_TABLE table=default.Table_d6e976e60b3e445492636b06856344c3 [id=0f32eabc16cd47bdb6addf705adfffa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.817857 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 (DEFAULT_TABLE table=default.Table_6e29e77c800340dbad41e6a6b3c1c0ec [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.818466 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3710b653aa24b7ca389aa95d7cdd2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.818653 4786 tablet_service.cc:1467] Processing CreateTablet for tablet a86ac17982324847949b8a9245329489 (DEFAULT_TABLE table=default.Table_e26f10cba82342b0a8719a897d1af394 [id=4dbb0ce32dad4373a8362c9297990767]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.818946 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.819232 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86ac17982324847949b8a9245329489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.819252 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (table default.Table_5db43ecab50c42d6bd7cea5b311d5f95 [id=5765715dd8e74bd5b16aa04cbc596b80]) successfully deleted
I20240503 02:12:24.821152 4462 leader_election.cc:302] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:24.823787 5106 tablet_bootstrap.cc:492] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:24.823872 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 038f11066dc34ba9a833f84a54ff20bf (DEFAULT_TABLE table=default.Table_3dce7a577f4f4969aa663ff5e3fb9234 [id=29fa2840604645a0b1a7bf9dc25bf919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.824097 5106 ts_tablet_manager.cc:1367] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.000s
I20240503 02:12:24.824311 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f11066dc34ba9a833f84a54ff20bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.825685 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.829005 4648 tablet_service.cc:1467] Processing CreateTablet for tablet a86ac17982324847949b8a9245329489 (DEFAULT_TABLE table=default.Table_e26f10cba82342b0a8719a897d1af394 [id=4dbb0ce32dad4373a8362c9297990767]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.825613 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 038f11066dc34ba9a833f84a54ff20bf (DEFAULT_TABLE table=default.Table_3dce7a577f4f4969aa663ff5e3fb9234 [id=29fa2840604645a0b1a7bf9dc25bf919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.829332 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.829736 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86ac17982324847949b8a9245329489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.833169 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f11066dc34ba9a833f84a54ff20bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.833388 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3710b653aa24b7ca389aa95d7cdd2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.840363 4509 tablet_service.cc:1467] Processing CreateTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c (DEFAULT_TABLE table=default.Table_e8a9dc27e1654f2e92a1bd71999f7959 [id=6d0dbeeacad04c1f8db9512b07d8a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.842566 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 288) took 2162 ms. Trace:
I20240503 02:12:24.842698 5159 rpcz_store.cc:276] 0503 02:12:22.680263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.680450 (+ 187us) service_pool.cc:225] Handling call
0503 02:12:24.755209 (+2074759us) ts_tablet_manager.cc:1120] Deleting tablet 894b753b71a04193905a190da320a260
0503 02:12:24.755220 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.774454 (+ 19234us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.777801 (+ 3347us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.842527 (+ 64726us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059916}
I20240503 02:12:24.843039 5159 tablet_replica.cc:330] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.843245 5159 raft_consensus.cc:2229] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.843461 5159 raft_consensus.cc:2258] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.844012 5159 ts_tablet_manager.cc:1875] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.847280 4652 tablet_service.cc:1467] Processing CreateTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c (DEFAULT_TABLE table=default.Table_e8a9dc27e1654f2e92a1bd71999f7959 [id=6d0dbeeacad04c1f8db9512b07d8a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.847752 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.848284 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 894b753b71a04193905a190da320a260 (table default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]) successfully deleted
I20240503 02:12:24.825212 5106 raft_consensus.cc:348] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.849411 5106 raft_consensus.cc:374] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:24.849519 5106 raft_consensus.cc:729] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:24.850809 5106 consensus_queue.cc:260] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.851331 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 20b6a18be64049049e1a083bf4af51ba (DEFAULT_TABLE table=default.Table_d372fe582377472fb9b1ae98f230a96f [id=03544a9ce3c043bb82e1e86f9fd29609]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.851467 5106 ts_tablet_manager.cc:1398] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:12:24.851878 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b6a18be64049049e1a083bf4af51ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.854007 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 20b6a18be64049049e1a083bf4af51ba (DEFAULT_TABLE table=default.Table_d372fe582377472fb9b1ae98f230a96f [id=03544a9ce3c043bb82e1e86f9fd29609]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.854853 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b6a18be64049049e1a083bf4af51ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.855160 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.854458 4786 tablet_service.cc:1467] Processing CreateTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c (DEFAULT_TABLE table=default.Table_e8a9dc27e1654f2e92a1bd71999f7959 [id=6d0dbeeacad04c1f8db9512b07d8a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.859547 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.859658 5286 raft_consensus.cc:2781] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.860520 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 20b6a18be64049049e1a083bf4af51ba (DEFAULT_TABLE table=default.Table_d372fe582377472fb9b1ae98f230a96f [id=03544a9ce3c043bb82e1e86f9fd29609]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.860976 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b6a18be64049049e1a083bf4af51ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.865202 5478 raft_consensus.cc:482] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.865379 5478 raft_consensus.cc:504] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.866153 5478 leader_election.cc:288] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.867107 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.867357 4811 raft_consensus.cc:2454] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.867333 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:24.867591 4528 raft_consensus.cc:2454] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.867830 4608 leader_election.cc:302] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.868218 5478 raft_consensus.cc:2781] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.868355 5478 raft_consensus.cc:482] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.868458 5478 raft_consensus.cc:3037] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.869837 5286 raft_consensus.cc:686] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:24.869939 5547 raft_consensus.cc:482] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.870129 5547 raft_consensus.cc:504] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.870183 5286 consensus_queue.cc:237] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.870945 5547 leader_election.cc:288] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.877040 5115 ts_tablet_manager.cc:1875] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.877461 5106 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:24.879185 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.880174 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:24.880275 4746 consensus_peers.cc:485] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387a596cfe654aa99e83d070b7884e6b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:24.880434 4811 raft_consensus.cc:2454] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.881099 5478 raft_consensus.cc:504] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.881423 5106 tablet_bootstrap.cc:654] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.881562 4673 raft_consensus.cc:2379] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:24.883949 5478 leader_election.cc:288] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.884500 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.884721 4811 raft_consensus.cc:3037] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.884756 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46560ac6d94943d2a8fb3b3904f2b5b6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:24.885071 4528 raft_consensus.cc:3037] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.885166 5159 ts_tablet_manager.cc:1888] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:24.885326 5159 log.cc:1196] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/cecaf785cb7e4aa1b6bef8d572e30706
I20240503 02:12:24.885843 5159 ts_tablet_manager.cc:1909] T cecaf785cb7e4aa1b6bef8d572e30706 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.886741 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 292) took 2200 ms. Trace:
I20240503 02:12:24.886843 5159 rpcz_store.cc:276] 0503 02:12:22.686228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.695497 (+ 9269us) service_pool.cc:225] Handling call
0503 02:12:24.842978 (+2147481us) ts_tablet_manager.cc:1120] Deleting tablet cecaf785cb7e4aa1b6bef8d572e30706
0503 02:12:24.842988 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.875228 (+ 32240us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.885035 (+ 9807us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.886670 (+ 1635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147263}
I20240503 02:12:24.887315 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 293) took 2200 ms. Trace:
I20240503 02:12:24.887415 5159 rpcz_store.cc:276] 0503 02:12:22.686614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.695735 (+ 9121us) service_pool.cc:225] Handling call
0503 02:12:24.887138 (+2191403us) ts_tablet_manager.cc:1120] Deleting tablet f2f9b962c9264f87b798df33544cb15d
0503 02:12:24.887149 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.887260 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2191218}
I20240503 02:12:24.887724 5159 tablet_replica.cc:330] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.887943 5159 raft_consensus.cc:2229] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.887988 4528 raft_consensus.cc:2454] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.887997 4811 raft_consensus.cc:2454] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.888223 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet cecaf785cb7e4aa1b6bef8d572e30706 (table default.Table_2243856948a74aaab070be52ee0905a2 [id=db26d21e021444d795aade9f022af228]) successfully deleted
I20240503 02:12:24.888327 5159 raft_consensus.cc:2258] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:24.888613 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f2f9b962c9264f87b798df33544cb15d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2f9b962c9264f87b798df33544cb15d already in progress: creating tablet
I20240503 02:12:24.888892 5547 raft_consensus.cc:482] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.889253 5547 raft_consensus.cc:504] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.890729 5547 leader_election.cc:288] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:24.890733 4462 leader_election.cc:302] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:24.891880 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53dad4e45a184dc1ae4a82021a48e9be" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.892105 4811 raft_consensus.cc:2454] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.892581 5547 raft_consensus.cc:482] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:24.893487 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65774b69a8bd4664a6b55b84eba37f7f consensus state has no leader
I20240503 02:12:24.893893 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cee118c15da24103b1ad3d2b2daf3f52" } modify_external_catalogs: true
I20240503 02:12:24.894104 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cee118c15da24103b1ad3d2b2daf3f52" } modify_external_catalogs: true
I20240503 02:12:24.894809 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_93262e3fdea24977a7a3aa557373b9eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.895382 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93262e3fdea24977a7a3aa557373b9eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.894192 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 (DEFAULT_TABLE table=default.Table_6e29e77c800340dbad41e6a6b3c1c0ec [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.897012 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.897420 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8372823b687d4ad1b174131ac0a070b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:24.897918 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8122f1bfb50244dca4a7836db5ef79bb" } modify_external_catalogs: true
I20240503 02:12:24.898080 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8122f1bfb50244dca4a7836db5ef79bb" } modify_external_catalogs: true
I20240503 02:12:24.899315 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2b80e65a3283458d8556b0832d236304" } modify_external_catalogs: true
I20240503 02:12:24.899482 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2b80e65a3283458d8556b0832d236304" } modify_external_catalogs: true
W20240503 02:12:24.901314 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8372823b687d4ad1b174131ac0a070b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.905099 5455 raft_consensus.cc:2741] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:12:24.905839 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]). Expected version 3 got 0
I20240503 02:12:24.906697 4499 tablet_service.cc:1467] Processing CreateTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe (DEFAULT_TABLE table=default.Table_d6e976e60b3e445492636b06856344c3 [id=0f32eabc16cd47bdb6addf705adfffa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.906997 4273 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_b43cd816d9de4772af938d5184d05ce4 [id=7048bc91b0b844f694937676dbc1026f] alter complete (version 1)
I20240503 02:12:24.907191 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3710b653aa24b7ca389aa95d7cdd2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.908810 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c5d56a6132f94935a81edadcf1f0d7f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.909561 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5d56a6132f94935a81edadcf1f0d7f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.910374 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3e6a3ca705e8482a8cd6a6fb17d93470
I20240503 02:12:24.910813 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 3e6a3ca705e8482a8cd6a6fb17d93470 deleted and replaced by tablet 04d555c8d8844ee49c960a5279485480
I20240503 02:12:24.911026 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 4b4972057fae45168756d7c28ddbe2bf from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:24.913769 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2152f205cf1b4ec7ae03605bbfa007e1
I20240503 02:12:24.914402 4270 catalog_manager.cc:6554] ReplaceTablet: tablet 2152f205cf1b4ec7ae03605bbfa007e1 deleted and replaced by tablet 0a92f97d40424c509088ae70ba84f3dd
I20240503 02:12:24.914573 4270 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 494) took 1181 ms. Trace:
I20240503 02:12:24.914700 4270 rpcz_store.cc:276] 0503 02:12:23.733165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.483127 (+749962us) service_pool.cc:225] Handling call
0503 02:12:24.486631 (+ 3504us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7b505b 00007fd3f17a6be7 00007fd3f17a6915 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:24.914491 (+427860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:24.805359 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:24.805452 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:24.805458 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:24.805460 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:24.805569 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:24.805580 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:24.805585 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:24.805666 (+ 81us) tablet.cc:699] Row locks acquired
0503 02:12:24.805668 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:24.805701 (+ 33us) write_op.cc:270] Start()
0503 02:12:24.805731 (+ 30us) write_op.cc:276] Timestamp: P: 1714702344805662 usec, L: 0
0503 02:12:24.805735 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:24.805933 (+ 198us) log.cc:842] Serialized 622 byte log entry
0503 02:12:24.902103 (+ 96170us) op_driver.cc:444] REPLICATION: finished
0503 02:12:24.902861 (+ 758us) write_op.cc:301] APPLY: starting
0503 02:12:24.902875 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:24.902917 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:24.902947 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:24.903051 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:24.903121 (+ 70us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:24.903132 (+ 11us) write_op.cc:312] APPLY: finished
0503 02:12:24.903189 (+ 57us) log.cc:842] Serialized 29 byte log entry
0503 02:12:24.903259 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:24.903277 (+ 18us) write_op.cc:454] Released schema lock
0503 02:12:24.903284 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:24.903341 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:24.903347 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":7028608,"child_traces":[["op",{"apply.queue_time_us":724,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":535,"num_ops":2,"prepare.queue_time_us":708,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":628,"replication_time_us":96337}]]}
I20240503 02:12:24.916203 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.916546 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c713f241c06a4a7c8106ba83bcab0c70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:24.917068 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c713f241c06a4a7c8106ba83bcab0c70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:24.921638 5159 ts_tablet_manager.cc:1875] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.924197 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_114d4d6efe8947578cff18d0c2f235d2" } modify_external_catalogs: true
I20240503 02:12:24.924340 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.924474 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d719c58974442281c7d694e1ca485e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.924480 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_114d4d6efe8947578cff18d0c2f235d2" } modify_external_catalogs: true
I20240503 02:12:24.925079 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.925594 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.926453 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b405182d426d419d9ba68b4159595b78" } new_table_name: "default.Table_e530bacca394443cbd9883735523abf1" modify_external_catalogs: true
I20240503 02:12:24.929050 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 038f11066dc34ba9a833f84a54ff20bf (DEFAULT_TABLE table=default.Table_3dce7a577f4f4969aa663ff5e3fb9234 [id=29fa2840604645a0b1a7bf9dc25bf919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.929663 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f11066dc34ba9a833f84a54ff20bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.933976 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a92f97d40424c509088ae70ba84f3dd) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:24.938673 4608 leader_election.cc:302] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.940403 5159 ts_tablet_manager.cc:1888] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:24.940513 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a92f97d40424c509088ae70ba84f3dd) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.940635 5159 log.cc:1196] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7d5e627d818245e0bdeb16531499f7d5
I20240503 02:12:24.941202 5159 ts_tablet_manager.cc:1909] T 7d5e627d818245e0bdeb16531499f7d5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.942831 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 894b753b71a04193905a190da320a260 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.943452 5114 tablet_replica.cc:330] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:24.943522 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 289) took 2263 ms. Trace:
I20240503 02:12:24.943775 5114 raft_consensus.cc:2229] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:24.943637 5159 rpcz_store.cc:276] 0503 02:12:22.680374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.680435 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:24.887641 (+2207206us) ts_tablet_manager.cc:1120] Deleting tablet 7d5e627d818245e0bdeb16531499f7d5
0503 02:12:24.887649 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.927235 (+ 39586us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.940227 (+ 12992us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.943482 (+ 3255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2191653}
I20240503 02:12:24.944110 5114 pending_rounds.cc:62] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:24.944213 5114 pending_rounds.cc:68] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420802977361920 op_type: NO_OP noop_request { }
I20240503 02:12:24.944378 5114 raft_consensus.cc:2866] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:24.944532 5114 raft_consensus.cc:2258] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.938642 5478 raft_consensus.cc:482] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.945842 5114 ts_tablet_manager.cc:1875] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.945880 5478 raft_consensus.cc:504] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.947991 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 291) took 2262 ms. Trace:
I20240503 02:12:24.948160 5159 rpcz_store.cc:276] 0503 02:12:22.685864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.685920 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:24.947834 (+2261914us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:24.947845 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.947957 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2242562}
I20240503 02:12:24.948716 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 290) took 2263 ms. Trace:
I20240503 02:12:24.948870 5159 rpcz_store.cc:276] 0503 02:12:22.685749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.685923 (+ 174us) service_pool.cc:225] Handling call
0503 02:12:24.948582 (+2262659us) ts_tablet_manager.cc:1120] Deleting tablet 88b173d78ba74ede801b12cccbeef6b1
0503 02:12:24.948591 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.948690 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235422}
I20240503 02:12:24.949851 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53dad4e45a184dc1ae4a82021a48e9be" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:24.950145 4671 raft_consensus.cc:2454] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:24.950541 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 295) took 2124 ms. Trace:
I20240503 02:12:24.950704 5159 rpcz_store.cc:276] 0503 02:12:22.825825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.825909 (+ 84us) service_pool.cc:225] Handling call
0503 02:12:24.950381 (+2124472us) ts_tablet_manager.cc:1120] Deleting tablet c1f39e8d17b44895bd5c979d82145ad9
0503 02:12:24.950390 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.950508 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2122773}
I20240503 02:12:24.951218 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 294) took 2147 ms. Trace:
I20240503 02:12:24.953855 5382 raft_consensus.cc:2781] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.954052 5382 raft_consensus.cc:686] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:24.954053 5159 rpcz_store.cc:276] 0503 02:12:22.807719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.808604 (+ 885us) service_pool.cc:225] Handling call
0503 02:12:24.951101 (+2142497us) ts_tablet_manager.cc:1120] Deleting tablet b080fc5ffc564420a0b7f2b7460ebbbb
0503 02:12:24.951110 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.951190 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121020}
I20240503 02:12:24.954797 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 296) took 2124 ms. Trace:
I20240503 02:12:24.955364 5159 rpcz_store.cc:276] 0503 02:12:22.830450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.830659 (+ 209us) service_pool.cc:225] Handling call
0503 02:12:24.954664 (+2124005us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:24.954673 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.954767 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123355}
I20240503 02:12:24.955855 5159 tablet_replica.cc:330] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:24.956029 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 50182.2us
I20240503 02:12:24.956130 5159 raft_consensus.cc:2229] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.956676 5159 raft_consensus.cc:2258] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.956780 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.957054 4811 raft_consensus.cc:2454] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:24.957355 5382 consensus_queue.cc:237] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.958591 5479 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.958858 5159 ts_tablet_manager.cc:1875] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:24.958752 5479 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.959936 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd402c0eb9e4cf49dc722f2dd214e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.961023 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c07c7cbb743d4a8e9610281a00f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.961254 5478 leader_election.cc:288] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.961519 4671 raft_consensus.cc:3037] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.962296 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.963148 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a92f97d40424c509088ae70ba84f3dd) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.963248 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 24737f8a8ca74996b9f54d49d12b9288 with delete_type TABLET_DATA_DELETED (replaced by tablet c2efdbfa909c475d95e99cf24206608f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.964013 5479 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.964195 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:24.966830 5159 ts_tablet_manager.cc:1888] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:24.962602 4608 leader_election.cc:302] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.969878 5479 raft_consensus.cc:2781] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.970192 5479 raft_consensus.cc:482] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.970299 5479 raft_consensus.cc:3037] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.972389 4721 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:24.979063 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a596cfe654aa99e83d070b7884e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.979660 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:24.981334 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6a3ca705e8482a8cd6a6fb17d93470 already in progress: creating tablet
I20240503 02:12:24.984079 5479 raft_consensus.cc:504] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.985558 5479 leader_election.cc:288] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.969554 5331 consensus_queue.cc:1035] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.970111 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:24.991858 4811 raft_consensus.cc:2379] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:24.992025 5331 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.975268 5426 raft_consensus.cc:482] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.986328 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:24.992213 5331 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.992323 4810 raft_consensus.cc:3037] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.992879 5331 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.993674 5159 log.cc:1196] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c823155b774e42a5a422994903eae167
I20240503 02:12:24.994310 5159 ts_tablet_manager.cc:1909] T c823155b774e42a5a422994903eae167 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:24.995169 4810 raft_consensus.cc:2454] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:24.992218 5426 raft_consensus.cc:504] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:24.995966 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet aa699a1dc3c34be59925560d5e11f49e
I20240503 02:12:24.996490 5426 leader_election.cc:288] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:24.995579 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 297) took 2162 ms. Trace:
I20240503 02:12:24.997387 5159 rpcz_store.cc:276] 0503 02:12:22.833408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.833472 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:24.955786 (+2122314us) ts_tablet_manager.cc:1120] Deleting tablet c823155b774e42a5a422994903eae167
0503 02:12:24.955795 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:24.963549 (+ 7754us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.966702 (+ 3153us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:24.995522 (+ 28820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121502}
I20240503 02:12:24.998296 5159 tablet_replica.cc:330] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:24.985976 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:24.997165 4608 leader_election.cc:302] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:24.999361 5479 raft_consensus.cc:2781] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.999370 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:24.999591 5479 raft_consensus.cc:686] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:24.999931 5479 consensus_queue.cc:237] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.000257 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ff8d2c0991e849b5afa35e7230db298d
I20240503 02:12:25.001374 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_15cc1a53bc594d6b8a04c4588fa6ec14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.002197 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15cc1a53bc594d6b8a04c4588fa6ec14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.004148 4272 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 0a92f97d40424c509088ae70ba84f3dd
I20240503 02:12:25.004446 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 19fb0929bd234788b5313d96d2e4845c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.005163 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_36b1bbc5ec49474881c8c6eca0df22ba" } new_table_name: "default.Table_3d65228f6c2b4694bb60f3118ad65af7" modify_external_catalogs: true
I20240503 02:12:25.006922 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_b405182d426d419d9ba68b4159595b78 [id=01ac062e9b344819ad99b55ae572916d]). Expected version 1 got 0
I20240503 02:12:25.007347 5159 raft_consensus.cc:2229] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:25.007750 5159 pending_rounds.cc:62] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:25.007877 5159 pending_rounds.cc:68] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420797291036672 op_type: NO_OP noop_request { }
I20240503 02:12:25.008066 5159 raft_consensus.cc:2866] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:25.008214 5159 raft_consensus.cc:2258] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.008905 5159 ts_tablet_manager.cc:1875] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.009588 5115 ts_tablet_manager.cc:1888] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.009831 5115 log.cc:1196] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c823155b774e42a5a422994903eae167
I20240503 02:12:25.007863 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a567b7af96934be8836963c85cc81473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.012763 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a567b7af96934be8836963c85cc81473 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.013309 5106 tablet_bootstrap.cc:492] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.013628 5106 ts_tablet_manager.cc:1367] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.141s user 0.015s sys 0.073s
W20240503 02:12:25.015169 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dedead5a2bf347f6991cce8e9b4db708 consensus state has no leader
I20240503 02:12:25.015059 5106 raft_consensus.cc:348] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.015507 5106 raft_consensus.cc:374] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.015579 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_df799d52e6434e0f90b831960d0f39c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.015650 5106 raft_consensus.cc:729] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:25.016077 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df799d52e6434e0f90b831960d0f39c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.008953 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_247b02731e084bc5985d5f3d2dbac760" } modify_external_catalogs: true
I20240503 02:12:25.017431 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_247b02731e084bc5985d5f3d2dbac760" } modify_external_catalogs: true
I20240503 02:12:25.009328 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4b4972057fae45168756d7c28ddbe2bf
I20240503 02:12:25.018560 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 4b4972057fae45168756d7c28ddbe2bf deleted and replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b
I20240503 02:12:25.019114 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8a90ebcf29004d1a925b69fa20998dd3
I20240503 02:12:25.024205 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.024194 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:25.024736 4746 leader_election.cc:341] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.024797 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:25.025034 4746 leader_election.cc:341] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.025173 4746 leader_election.cc:302] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:25.025624 4746 leader_election.cc:341] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.025846 5553 raft_consensus.cc:2726] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.028404 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:25.028986 4746 leader_election.cc:341] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.029120 4746 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:25.011154 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0a16926a4afd4923bed405fe42c0a348"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.029647 5553 raft_consensus.cc:2726] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:25.030297 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a16926a4afd4923bed405fe42c0a348 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.031772 5115 ts_tablet_manager.cc:1909] T c823155b774e42a5a422994903eae167 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:25.016158 5106 consensus_queue.cc:260] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.036670 5382 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.036849 5382 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.037431 5159 ts_tablet_manager.cc:1888] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.037559 5382 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.037592 5159 log.cc:1196] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/003c9c3e0a1d400c94bfaefada2e9faa
I20240503 02:12:25.038064 5159 ts_tablet_manager.cc:1909] T 003c9c3e0a1d400c94bfaefada2e9faa P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.038919 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.041337 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d510e8f3fde248f29d85de95822d83af" } new_table_name: "default.Table_c3878b4f48444a1cad1c1a9eae99dd33" modify_external_catalogs: true
W20240503 02:12:25.036998 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd66f88a24b74f2c9b2050af081d4601 consensus state has no leader
I20240503 02:12:25.039155 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0134301a980f41e5847132881085032e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.042668 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0134301a980f41e5847132881085032e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.042281 5106 ts_tablet_manager.cc:1398] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240503 02:12:25.039558 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1b852a3367de45db9bba22405e5e3bac" } new_table_name: "default.Table_fd7702f08f474fd5beb82be6891ac6f8" modify_external_catalogs: true
I20240503 02:12:25.046983 5103 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:25.048007 5105 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.048238 5105 ts_tablet_manager.cc:1367] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.333s user 0.005s sys 0.005s
W20240503 02:12:24.988248 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2152f205cf1b4ec7ae03605bbfa007e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2152f205cf1b4ec7ae03605bbfa007e1 already in progress: creating tablet
I20240503 02:12:25.049093 5105 raft_consensus.cc:348] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.049326 5105 raft_consensus.cc:374] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.049400 5105 raft_consensus.cc:729] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:25.049531 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 61272.5us
I20240503 02:12:25.049738 5105 consensus_queue.cc:260] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.050290 5105 ts_tablet_manager.cc:1398] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:25.051995 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.052237 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet aa699a1dc3c34be59925560d5e11f49e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.055703 4859 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:25.048038 5547 raft_consensus.cc:504] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.047035 5106 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.060613 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.060900 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.061064 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.055977 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.059193 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet aa699a1dc3c34be59925560d5e11f49e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.061569 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.061511 5115 tablet_replica.cc:330] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.059383 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.062336 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.062381 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet b28bb9ac773c4bc685cb8193e574460b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.063936 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bfb8401dc518472abd4e26a2c788377d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.064725 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfb8401dc518472abd4e26a2c788377d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.065296 5105 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:25.066624 5115 raft_consensus.cc:2229] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.069617 5115 raft_consensus.cc:2258] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.070647 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.071993 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 298) took 2237 ms. Trace:
I20240503 02:12:25.072161 5159 rpcz_store.cc:276] 0503 02:12:22.834400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.836113 (+ 1713us) service_pool.cc:225] Handling call
0503 02:12:24.998208 (+2162095us) ts_tablet_manager.cc:1120] Deleting tablet 003c9c3e0a1d400c94bfaefada2e9faa
0503 02:12:24.998218 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.012620 (+ 14402us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.037285 (+ 24665us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.071935 (+ 34650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2161896}
I20240503 02:12:25.074072 5115 ts_tablet_manager.cc:1875] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.078678 5105 tablet_bootstrap.cc:654] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.078775 5106 tablet_bootstrap.cc:654] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.079282 5159 tablet_replica.cc:330] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.079622 5159 raft_consensus.cc:2229] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.079893 5159 raft_consensus.cc:2258] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.088708 5159 ts_tablet_manager.cc:1875] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.090162 5115 ts_tablet_manager.cc:1888] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.090355 5115 log.cc:1196] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/24737f8a8ca74996b9f54d49d12b9288
I20240503 02:12:25.090751 5115 ts_tablet_manager.cc:1909] T 24737f8a8ca74996b9f54d49d12b9288 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:25.098088 5106 tablet_bootstrap.cc:492] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.098352 5106 ts_tablet_manager.cc:1367] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.003s
I20240503 02:12:25.099295 5106 raft_consensus.cc:348] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.099299 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_150d9a5a76ef45ba89eae071bd04eab6" } new_table_name: "default.Table_29fbbe157bc0404fa71f43052d345a47" modify_external_catalogs: true
I20240503 02:12:25.099520 5106 raft_consensus.cc:374] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.099646 5106 raft_consensus.cc:729] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.099934 5106 consensus_queue.cc:260] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.100669 5106 ts_tablet_manager.cc:1398] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:25.100983 5106 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:25.101330 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f630bf6a5740758fa1f81477ec59d3 consensus state has no leader
I20240503 02:12:25.102542 5106 tablet_bootstrap.cc:654] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.106364 5115 tablet_replica.cc:330] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.106570 5115 raft_consensus.cc:2229] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.106762 5115 raft_consensus.cc:2258] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.107867 5159 ts_tablet_manager.cc:1888] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:25.111740 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607e49cde2fd4aab9f44921d3c3921a5 consensus state has no leader
I20240503 02:12:25.111968 5159 log.cc:1196] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7d1210a1217247b9801226055e2f2195
I20240503 02:12:25.112524 5159 ts_tablet_manager.cc:1909] T 7d1210a1217247b9801226055e2f2195 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.112862 5115 ts_tablet_manager.cc:1875] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.121994 5106 tablet_bootstrap.cc:492] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.122225 5106 ts_tablet_manager.cc:1367] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 02:12:25.122750 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 299) took 2286 ms. Trace:
I20240503 02:12:25.122947 5159 rpcz_store.cc:276] 0503 02:12:22.836472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.836510 (+ 38us) service_pool.cc:225] Handling call
0503 02:12:25.072770 (+2236260us) ts_tablet_manager.cc:1120] Deleting tablet 7d1210a1217247b9801226055e2f2195
0503 02:12:25.072783 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.099453 (+ 26670us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.107708 (+ 8255us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.122705 (+ 14997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2236014}
I20240503 02:12:25.122993 5106 raft_consensus.cc:348] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.123240 5106 raft_consensus.cc:374] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.123323 5106 raft_consensus.cc:729] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.123620 5106 consensus_queue.cc:260] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.124336 5106 ts_tablet_manager.cc:1398] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:25.124646 5106 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.127414 5159 tablet_replica.cc:330] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.134752 5479 raft_consensus.cc:482] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.134986 5479 raft_consensus.cc:504] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.135818 5479 leader_election.cc:288] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.136600 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.136942 4810 raft_consensus.cc:2454] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:25.137508 4608 leader_election.cc:302] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:25.137825 5479 raft_consensus.cc:2781] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.137930 5479 raft_consensus.cc:482] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.138044 5479 raft_consensus.cc:3037] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.139480 5105 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.139776 5105 ts_tablet_manager.cc:1367] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.025s
I20240503 02:12:25.141470 5479 raft_consensus.cc:504] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:25.142055 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 70549.9us
I20240503 02:12:25.142506 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c823155b774e42a5a422994903eae167 (table default.Table_7f7a688514414e1cbb2d14173f3d1976 [id=7abc43f4f4eb42828defca24da288002]) successfully deleted
I20240503 02:12:25.142532 5479 leader_election.cc:288] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:25.142921 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 34eb69d5f4bd427a8e430a00fb076f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34eb69d5f4bd427a8e430a00fb076f83 already in progress: creating tablet
W20240503 02:12:25.143230 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 27c13f257eae44689af07cc31ea254c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27c13f257eae44689af07cc31ea254c4 already in progress: creating tablet
I20240503 02:12:25.143640 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 24737f8a8ca74996b9f54d49d12b9288 (table default.Table_a7065a46b193410db33240f20db15636 [id=7053a43b0a61498599ad2ad9d6a06dfa]) successfully deleted
W20240503 02:12:25.143852 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet aa67857b858c4719b498291c52b7aceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67857b858c4719b498291c52b7aceb already in progress: creating tablet
I20240503 02:12:25.144237 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7d5e627d818245e0bdeb16531499f7d5 (table default.Table_3ae28439fd6b46438f67fd812635c2f4 [id=229a92f5222d41d68a3843dc1c8ccded]) successfully deleted
W20240503 02:12:25.144556 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 396b48865e954d05b9ba8084a2e7f29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 396b48865e954d05b9ba8084a2e7f29b already in progress: creating tablet
W20240503 02:12:25.144771 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 88b173d78ba74ede801b12cccbeef6b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88b173d78ba74ede801b12cccbeef6b1 already in progress: creating tablet
W20240503 02:12:25.144973 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c1f39e8d17b44895bd5c979d82145ad9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f39e8d17b44895bd5c979d82145ad9 already in progress: creating tablet
W20240503 02:12:25.145188 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b080fc5ffc564420a0b7f2b7460ebbbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b080fc5ffc564420a0b7f2b7460ebbbb already in progress: creating tablet
W20240503 02:12:25.145392 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f146f7cce0b24d3087bfe0973e2b7ef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f146f7cce0b24d3087bfe0973e2b7ef0 already in progress: creating tablet
I20240503 02:12:25.145591 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c823155b774e42a5a422994903eae167 (table default.Table_7f7a688514414e1cbb2d14173f3d1976 [id=7abc43f4f4eb42828defca24da288002]) successfully deleted
I20240503 02:12:25.145762 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 003c9c3e0a1d400c94bfaefada2e9faa (table default.Table_ec8c0cdf74694faba53463bb1c5a4b0a [id=5cac9c6682b64e30a1ae1ad81e094fba]) successfully deleted
I20240503 02:12:25.145939 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7d1210a1217247b9801226055e2f2195 (table default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]) successfully deleted
I20240503 02:12:25.147871 5106 tablet_bootstrap.cc:654] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.148178 5159 raft_consensus.cc:2229] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:25.148810 5159 raft_consensus.cc:2258] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.152007 5114 ts_tablet_manager.cc:1888] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.152195 5114 log.cc:1196] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/894b753b71a04193905a190da320a260
I20240503 02:12:25.152284 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet aa699a1dc3c34be59925560d5e11f49e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.153092 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:25.153461 5103 ts_tablet_manager.cc:1367] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.334s user 0.031s sys 0.055s
I20240503 02:12:25.153506 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.154587 4810 raft_consensus.cc:3037] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.154614 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.154480 5105 raft_consensus.cc:348] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.154774 5105 raft_consensus.cc:374] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.154942 5105 raft_consensus.cc:729] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.155184 5105 consensus_queue.cc:260] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.155534 5286 raft_consensus.cc:482] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.155730 5115 ts_tablet_manager.cc:1888] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.155683 5286 raft_consensus.cc:504] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.155854 5115 log.cc:1196] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/bf9c07c7cbb743d4a8e9610281a00f94
I20240503 02:12:25.156230 5115 ts_tablet_manager.cc:1909] T bf9c07c7cbb743d4a8e9610281a00f94 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:25.158110 5286 leader_election.cc:288] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.158282 5547 leader_election.cc:288] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.158476 5114 ts_tablet_manager.cc:1909] T 894b753b71a04193905a190da320a260 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.159490 5114 tablet_replica.cc:330] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.159759 5114 raft_consensus.cc:2229] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.159919 5114 raft_consensus.cc:2258] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.160012 4810 raft_consensus.cc:2454] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:25.160465 5159 ts_tablet_manager.cc:1875] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.160502 5114 ts_tablet_manager.cc:1875] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.160744 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.161568 4608 leader_election.cc:302] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:25.162585 5479 raft_consensus.cc:2781] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.162961 5479 raft_consensus.cc:686] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:25.163341 5479 consensus_queue.cc:237] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.165710 5105 ts_tablet_manager.cc:1398] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:12:25.168741 5159 ts_tablet_manager.cc:1888] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.170107 5105 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:25.179013 5105 tablet_bootstrap.cc:654] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.181326 5159 log.cc:1196] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/387a596cfe654aa99e83d070b7884e6b
I20240503 02:12:25.182219 5159 ts_tablet_manager.cc:1909] T 387a596cfe654aa99e83d070b7884e6b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.183079 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 300) took 2345 ms. Trace:
I20240503 02:12:25.183279 5159 rpcz_store.cc:276] 0503 02:12:22.837097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.837148 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:25.127338 (+2290190us) ts_tablet_manager.cc:1120] Deleting tablet 387a596cfe654aa99e83d070b7884e6b
0503 02:12:25.127350 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.148529 (+ 21179us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x6160001c0354. stack: 00007fced75a0756 00007fced75a0510 00007fced65d348d 0000000000955804 00000000009557cd 000000000095519a 00007fcee2e0ea00 00007fcee40e8b83 00007fcee6b7c57c 00007fcee6bae8d4 00007fcee6b8fbc0 00007fcee6b8f94d 0000000000a60bcc 00007fced75e6b74 00007fced75ea19b 00007fced75ea018
0503 02:12:25.164012 (+ 15483us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.168590 (+ 4578us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.183017 (+ 14427us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1907,"spinlock_wait_cycles":31862656,"tablet-delete.queue_time_us":2290000}
I20240503 02:12:25.184103 5159 tablet_replica.cc:330] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.184541 5159 raft_consensus.cc:2229] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.189817 5115 tablet_replica.cc:330] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.190137 5115 raft_consensus.cc:2229] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.190438 5115 raft_consensus.cc:2258] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.191378 5115 ts_tablet_manager.cc:1875] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:25.198644 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c2a8190210439fb7589cb02ec82bc2 consensus state has no leader
I20240503 02:12:25.200585 5114 ts_tablet_manager.cc:1888] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.200942 5114 log.cc:1196] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:25.202764 5159 raft_consensus.cc:2258] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.203433 5159 ts_tablet_manager.cc:1875] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.204129 5115 ts_tablet_manager.cc:1888] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.168052 5103 raft_consensus.cc:348] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.209430 5103 raft_consensus.cc:374] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.209712 5103 raft_consensus.cc:729] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.210275 5103 consensus_queue.cc:260] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.211277 5115 log.cc:1196] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/387a596cfe654aa99e83d070b7884e6b
I20240503 02:12:25.214030 5115 ts_tablet_manager.cc:1909] T 387a596cfe654aa99e83d070b7884e6b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:25.219771 5114 ts_tablet_manager.cc:1909] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.221563 5114 tablet_replica.cc:330] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.222059 5159 ts_tablet_manager.cc:1888] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.222081 5114 raft_consensus.cc:2229] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:25.222734 5114 pending_rounds.cc:62] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:25.222884 5114 pending_rounds.cc:68] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420804590690304 op_type: NO_OP noop_request { }
I20240503 02:12:25.223227 5114 raft_consensus.cc:2866] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:25.223493 5114 raft_consensus.cc:2258] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.224642 5114 ts_tablet_manager.cc:1875] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.225467 5159 log.cc:1196] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:25.225929 5159 ts_tablet_manager.cc:1909] T 6cd402c0eb9e4cf49dc722f2dd214e09 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.231877 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 301) took 2393 ms. Trace:
I20240503 02:12:25.232507 5103 ts_tablet_manager.cc:1398] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20240503 02:12:25.234767 5159 rpcz_store.cc:276] 0503 02:12:22.838137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.838186 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:25.184010 (+2345824us) ts_tablet_manager.cc:1120] Deleting tablet 6cd402c0eb9e4cf49dc722f2dd214e09
0503 02:12:25.184020 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.218403 (+ 34383us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.221895 (+ 3492us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.231839 (+ 9944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2345613}
I20240503 02:12:25.235924 5159 tablet_replica.cc:330] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.238847 5103 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:25.241153 5159 raft_consensus.cc:2229] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.241714 5159 raft_consensus.cc:2258] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.242550 5159 ts_tablet_manager.cc:1875] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.244093 5114 ts_tablet_manager.cc:1888] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.244287 5114 log.cc:1196] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/bf9c07c7cbb743d4a8e9610281a00f94
W20240503 02:12:25.244433 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 353676us
I20240503 02:12:25.244872 5103 tablet_bootstrap.cc:654] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.245427 4671 raft_consensus.cc:3037] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.255180 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 43119471f91c4a1ba900f267c88db9a6 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.255445 5105 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.255626 5105 ts_tablet_manager.cc:1367] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.008s
I20240503 02:12:25.255769 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_de35f554f3f64611a5a03c54e5c17e04" } modify_external_catalogs: true
I20240503 02:12:25.255980 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_de35f554f3f64611a5a03c54e5c17e04" } modify_external_catalogs: true
I20240503 02:12:25.256363 5105 raft_consensus.cc:348] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.256587 5105 raft_consensus.cc:374] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.256654 5105 raft_consensus.cc:729] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.256631 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet bf9c07c7cbb743d4a8e9610281a00f94 (table default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]) successfully deleted
I20240503 02:12:25.256896 5105 consensus_queue.cc:260] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:25.257086 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 87713e899d2148308316d807a6d76921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87713e899d2148308316d807a6d76921 already in progress: creating tablet
I20240503 02:12:25.257478 5105 ts_tablet_manager.cc:1398] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240503 02:12:25.257472 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9ed525a61c734ee2b23d5d4f319a7d67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed525a61c734ee2b23d5d4f319a7d67 already in progress: creating tablet
W20240503 02:12:25.257828 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet e94031465ce7407dab0f90ad8809ac21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94031465ce7407dab0f90ad8809ac21 already in progress: creating tablet
W20240503 02:12:25.258208 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d607da9689c14e51add9178a48b35913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d607da9689c14e51add9178a48b35913 already in progress: creating tablet
I20240503 02:12:25.258245 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_634134e8f4a14d2a86f04be0d642a853" } modify_external_catalogs: true
I20240503 02:12:25.258415 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_634134e8f4a14d2a86f04be0d642a853" } modify_external_catalogs: true
I20240503 02:12:25.258744 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 387a596cfe654aa99e83d070b7884e6b (table default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]) successfully deleted
I20240503 02:12:25.258972 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_af6ff741f769452798eade9274479e3d" } modify_external_catalogs: true
I20240503 02:12:25.259060 5105 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:25.259089 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7d1210a1217247b9801226055e2f2195 because the tablet was not found. No further retry: Not found: Tablet not found: 7d1210a1217247b9801226055e2f2195
I20240503 02:12:25.259171 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_af6ff741f769452798eade9274479e3d" } modify_external_catalogs: true
W20240503 02:12:25.259526 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6a3ca705e8482a8cd6a6fb17d93470 already in progress: creating tablet
I20240503 02:12:25.259783 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5916ed7fb62a4cdc910846e24178b911"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.259891 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ff8d2c0991e849b5afa35e7230db298d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8d2c0991e849b5afa35e7230db298d already in progress: creating tablet
W20240503 02:12:25.260351 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5916ed7fb62a4cdc910846e24178b911 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.261924 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2771f168295e4bcb84010eb642198fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.262202 5105 tablet_bootstrap.cc:654] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:25.262535 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2771f168295e4bcb84010eb642198fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.260226 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet aa699a1dc3c34be59925560d5e11f49e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa699a1dc3c34be59925560d5e11f49e already in progress: creating tablet
W20240503 02:12:25.278585 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4b4972057fae45168756d7c28ddbe2bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4972057fae45168756d7c28ddbe2bf already in progress: creating tablet
W20240503 02:12:25.278872 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8a90ebcf29004d1a925b69fa20998dd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a90ebcf29004d1a925b69fa20998dd3 already in progress: creating tablet
I20240503 02:12:25.279058 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 387a596cfe654aa99e83d070b7884e6b (table default.Table_4eb187ba818c4f67a55aaeb3f44caae6 [id=dd51d963965f4d9a88f2b1749ad1457d]) successfully deleted
I20240503 02:12:25.279255 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (table default.Table_5db43ecab50c42d6bd7cea5b311d5f95 [id=5765715dd8e74bd5b16aa04cbc596b80]) successfully deleted
I20240503 02:12:25.279423 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 894b753b71a04193905a190da320a260 (table default.Table_be08a6b601e14feb9ab4d23eb62484a6 [id=f4617e089803496aaf3d7c607adb094d]) successfully deleted
I20240503 02:12:25.279436 5159 ts_tablet_manager.cc:1888] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.279621 5159 log.cc:1196] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/53dad4e45a184dc1ae4a82021a48e9be
I20240503 02:12:25.279656 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 6cd402c0eb9e4cf49dc722f2dd214e09 (table default.Table_5db43ecab50c42d6bd7cea5b311d5f95 [id=5765715dd8e74bd5b16aa04cbc596b80]) successfully deleted
I20240503 02:12:25.284536 5159 ts_tablet_manager.cc:1909] T 53dad4e45a184dc1ae4a82021a48e9be P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.286092 5105 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.286341 5105 ts_tablet_manager.cc:1367] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20240503 02:12:25.255080 4671 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.257706 5106 tablet_bootstrap.cc:492] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.290351 5106 ts_tablet_manager.cc:1367] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.166s user 0.010s sys 0.018s
I20240503 02:12:25.291249 5106 raft_consensus.cc:348] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.291492 5106 raft_consensus.cc:374] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.292680 5106 raft_consensus.cc:729] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.292963 5106 consensus_queue.cc:260] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.293468 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 305) took 2303 ms. Trace:
I20240503 02:12:25.299012 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ab30b57f7d9044fca1bf76dc5e2bc323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.299837 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab30b57f7d9044fca1bf76dc5e2bc323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.301029 5159 rpcz_store.cc:276] 0503 02:12:22.989766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:22.989866 (+ 100us) service_pool.cc:225] Handling call
0503 02:12:25.235857 (+2245991us) ts_tablet_manager.cc:1120] Deleting tablet 53dad4e45a184dc1ae4a82021a48e9be
0503 02:12:25.235869 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.269359 (+ 33490us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.279304 (+ 9945us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.293406 (+ 14102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245676}
I20240503 02:12:25.301688 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 53dad4e45a184dc1ae4a82021a48e9be (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
I20240503 02:12:25.293991 5105 raft_consensus.cc:348] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.302055 5105 raft_consensus.cc:374] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.302227 5105 raft_consensus.cc:729] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.302644 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8372823b687d4ad1b174131ac0a070b8" } new_table_name: "default.Table_5b3b2e6e211c47338aa5ddca26669a20" modify_external_catalogs: true
I20240503 02:12:25.302553 5105 consensus_queue.cc:260] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.303251 5105 ts_tablet_manager.cc:1398] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:12:25.303764 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 43119471f91c4a1ba900f267c88db9a6
I20240503 02:12:25.304235 4268 catalog_manager.cc:6554] ReplaceTablet: tablet 43119471f91c4a1ba900f267c88db9a6 deleted and replaced by tablet 368aabce19984d03aa7116c8df990ef1
I20240503 02:12:25.304566 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_63229f7088d0478b9de9fa953f84866b" } modify_external_catalogs: true
I20240503 02:12:25.304772 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_63229f7088d0478b9de9fa953f84866b" } modify_external_catalogs: true
I20240503 02:12:25.305637 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.308454 5105 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:25.314270 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:25.310474 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 307) took 2280 ms. Trace:
I20240503 02:12:25.314707 5159 rpcz_store.cc:276] 0503 02:12:23.029735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.029779 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:25.310278 (+2280499us) ts_tablet_manager.cc:1120] Deleting tablet b9511e4de6c942b9b9e0790c88d3d543
0503 02:12:25.310288 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.310413 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280245}
W20240503 02:12:25.315335 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b9511e4de6c942b9b9e0790c88d3d543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9511e4de6c942b9b9e0790c88d3d543 already in progress: creating tablet
I20240503 02:12:25.315532 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_140dd34e051a4448bfeeafb111265836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.316067 5105 tablet_bootstrap.cc:654] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:25.316252 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_140dd34e051a4448bfeeafb111265836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.317318 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 308) took 2287 ms. Trace:
I20240503 02:12:25.317471 5159 rpcz_store.cc:276] 0503 02:12:23.030240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.030280 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:25.316233 (+2285953us) ts_tablet_manager.cc:1120] Deleting tablet 79884b3e73904c708d0d48bca85351b7
0503 02:12:25.316242 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.317265 (+ 1023us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2285781}
W20240503 02:12:25.318001 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 79884b3e73904c708d0d48bca85351b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79884b3e73904c708d0d48bca85351b7 already in progress: creating tablet
I20240503 02:12:25.318482 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 306) took 2288 ms. Trace:
I20240503 02:12:25.318619 5159 rpcz_store.cc:276] 0503 02:12:23.029594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.030554 (+ 960us) service_pool.cc:225] Handling call
0503 02:12:25.318317 (+2287763us) ts_tablet_manager.cc:1120] Deleting tablet 41ef3c6e5e7d40249b112144975ba220
0503 02:12:25.318326 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.318419 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2287596}
W20240503 02:12:25.319123 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 41ef3c6e5e7d40249b112144975ba220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41ef3c6e5e7d40249b112144975ba220 already in progress: creating tablet
I20240503 02:12:25.319594 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 309) took 2253 ms. Trace:
I20240503 02:12:25.319751 5159 rpcz_store.cc:276] 0503 02:12:23.066487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.066550 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:25.319397 (+2252847us) ts_tablet_manager.cc:1120] Deleting tablet 7394c9d374404fb199ab6ff170d7febe
0503 02:12:25.319406 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.319516 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252600}
W20240503 02:12:25.320230 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7394c9d374404fb199ab6ff170d7febe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7394c9d374404fb199ab6ff170d7febe already in progress: creating tablet
I20240503 02:12:25.320650 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 310) took 2252 ms. Trace:
I20240503 02:12:25.320778 5159 rpcz_store.cc:276] 0503 02:12:23.068353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.068409 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:25.320510 (+2252101us) ts_tablet_manager.cc:1120] Deleting tablet 53b149e66a0446eeade190dd7f5d2bec
0503 02:12:25.320518 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.320599 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2251892}
W20240503 02:12:25.321259 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 53b149e66a0446eeade190dd7f5d2bec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b149e66a0446eeade190dd7f5d2bec already in progress: creating tablet
I20240503 02:12:25.322073 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 321) took 2163 ms. Trace:
I20240503 02:12:25.322232 5159 rpcz_store.cc:276] 0503 02:12:23.159005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.162037 (+ 3032us) service_pool.cc:225] Handling call
0503 02:12:25.321518 (+2159481us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:25.321526 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.322021 (+ 495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2156511}
W20240503 02:12:25.322763 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 159ea1eb32394b04b72258d24196202b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea1eb32394b04b72258d24196202b already in progress: creating tablet
I20240503 02:12:25.323185 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 327) took 2117 ms. Trace:
I20240503 02:12:25.323318 5159 rpcz_store.cc:276] 0503 02:12:23.206184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.206239 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:25.323028 (+2116789us) ts_tablet_manager.cc:1120] Deleting tablet 66ca08877cd948e1b04736b54bc779db
0503 02:12:25.323037 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.323124 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116589}
W20240503 02:12:25.323776 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 66ca08877cd948e1b04736b54bc779db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66ca08877cd948e1b04736b54bc779db already in progress: creating tablet
I20240503 02:12:25.324193 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 328) took 2084 ms. Trace:
I20240503 02:12:25.324329 5159 rpcz_store.cc:276] 0503 02:12:23.239553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.239883 (+ 330us) service_pool.cc:225] Handling call
0503 02:12:25.324039 (+2084156us) ts_tablet_manager.cc:1120] Deleting tablet 9327a9e5472b4eda854320c3a95f8e56
0503 02:12:25.324047 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.324136 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2082234}
W20240503 02:12:25.324785 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9327a9e5472b4eda854320c3a95f8e56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9327a9e5472b4eda854320c3a95f8e56 already in progress: creating tablet
I20240503 02:12:25.325141 5159 tablet_replica.cc:330] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.325423 5159 raft_consensus.cc:2229] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.328775 4272 catalog_manager.cc:5484] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.329483 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]). Expected version 3 got 0
I20240503 02:12:25.329948 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
I20240503 02:12:25.330209 4272 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:25.331580 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:25.331880 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:25.335762 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5dbbfa2d1a49400c809d5b476036cdcd
I20240503 02:12:25.336613 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a2295477032745c3b7b2839c77679705"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.337109 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:25.337231 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2295477032745c3b7b2839c77679705 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.338755 4792 tablet_service.cc:1467] Processing CreateTablet for tablet bca69a5b48754602b04481fb6a94864f (DEFAULT_TABLE table=default.Table_1b6529a1250745eea5d10a3f304a28e7 [id=2328ebc7628741a3b60122d3669d6245]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.339190 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca69a5b48754602b04481fb6a94864f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:25.339394 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a047360ed8843008c4ba633a65b2769 has no consensus state
I20240503 02:12:25.339859 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 905ca3be01e344d38350a58ac9f0d321 (DEFAULT_TABLE table=default.Table_d47e1a80afa040948b01f4ba1d3bf10b [id=74f1b122b8b3437b86a055dbb1222030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.339851 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c8707ebf10c24701a50991c2b06b50bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.340269 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905ca3be01e344d38350a58ac9f0d321. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:25.340366 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8707ebf10c24701a50991c2b06b50bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.342207 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b1056f446ede4a8c9a87907ab91da286" } new_table_name: "default.Table_f1c7b1c385fa46b0a311b8957066d4eb" modify_external_catalogs: true
I20240503 02:12:25.347163 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 76b01907833d44f6a000a4de3a2a6e96
I20240503 02:12:25.350510 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.348212 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6d070a69982c45b7904a12c6df0386b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.350872 4673 raft_consensus.cc:3037] T c63499fb112c48238d6b84110e394779 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:25.351234 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d070a69982c45b7904a12c6df0386b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.349697 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 84b6e303e34a465caffed6c9e1e6bf74
I20240503 02:12:25.353353 5159 raft_consensus.cc:2258] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.353928 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet b91abaf51abb4c2ca65412c3771c1ef2 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.354678 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_029aae84b86047689e44978da354aefa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.355422 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_029aae84b86047689e44978da354aefa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.358438 5106 ts_tablet_manager.cc:1398] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:25.358839 5159 ts_tablet_manager.cc:1875] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.358901 5106 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.362615 4649 tablet_service.cc:1467] Processing CreateTablet for tablet bca69a5b48754602b04481fb6a94864f (DEFAULT_TABLE table=default.Table_1b6529a1250745eea5d10a3f304a28e7 [id=2328ebc7628741a3b60122d3669d6245]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.362870 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 905ca3be01e344d38350a58ac9f0d321 (DEFAULT_TABLE table=default.Table_d47e1a80afa040948b01f4ba1d3bf10b [id=74f1b122b8b3437b86a055dbb1222030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.363042 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.363250 4652 tablet_service.cc:1467] Processing CreateTablet for tablet ed4e84a639b2403082b737d073f00ad7 (DEFAULT_TABLE table=default.Table_6616d554610d429190adcee205cf086c [id=3cb9bed8bee34911aab76b39eaa68dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.364686 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dce5dc7647cd4035b6cb74f11e45c718" } new_table_name: "default.Table_c7106f5dd350475781a5c67d7c4a6b30" modify_external_catalogs: true
I20240503 02:12:25.365885 5426 raft_consensus.cc:482] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.366055 5426 raft_consensus.cc:504] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.366652 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.366721 5426 leader_election.cc:288] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.366935 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.367820 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca69a5b48754602b04481fb6a94864f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:25.367775 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5dbbfa2d1a49400c809d5b476036cdcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dbbfa2d1a49400c809d5b476036cdcd already in progress: creating tablet
W20240503 02:12:25.368072 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 43119471f91c4a1ba900f267c88db9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43119471f91c4a1ba900f267c88db9a6 already in progress: creating tablet
W20240503 02:12:25.368328 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 84b6e303e34a465caffed6c9e1e6bf74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b6e303e34a465caffed6c9e1e6bf74 already in progress: creating tablet
W20240503 02:12:25.368553 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 76b01907833d44f6a000a4de3a2a6e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76b01907833d44f6a000a4de3a2a6e96 already in progress: creating tablet
I20240503 02:12:25.369223 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.369642 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.370330 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905ca3be01e344d38350a58ac9f0d321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.371187 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:25.372109 4746 leader_election.cc:341] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.373224 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4e84a639b2403082b737d073f00ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.376103 5106 tablet_bootstrap.cc:654] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.379215 4786 tablet_service.cc:1467] Processing CreateTablet for tablet ed4e84a639b2403082b737d073f00ad7 (DEFAULT_TABLE table=default.Table_6616d554610d429190adcee205cf086c [id=3cb9bed8bee34911aab76b39eaa68dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.379892 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4e84a639b2403082b737d073f00ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.381523 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8d0494ee4bd746069bfaf39bfb46687d
I20240503 02:12:25.382305 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0ad200f18fbc4fa3883ee56709b508ec" } new_table_name: "default.Table_1a50f808193c474c97646ae5b9ab2a0c" modify_external_catalogs: true
I20240503 02:12:25.384203 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:25.385231 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8d0494ee4bd746069bfaf39bfb46687d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d0494ee4bd746069bfaf39bfb46687d already in progress: creating tablet
I20240503 02:12:25.385731 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.390093 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a88710ac9062472f80bf6fcfb82793a0
I20240503 02:12:25.390990 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_55379d2e66cd42d28abf6943d13ca28e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.391561 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55379d2e66cd42d28abf6943d13ca28e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.397205 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.397233 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f (DEFAULT_TABLE table=default.Table_7ea27632721644359e27e0917381fb1c [id=5551c6d493d74bc39cea10e5a8198e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.397687 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b3d1e7ce83425db0bd56457d9f2544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.398629 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b91a3f3cad74f6aa6035a013cca651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.405059 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet a88710ac9062472f80bf6fcfb82793a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.409632 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet a88710ac9062472f80bf6fcfb82793a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.410333 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f (DEFAULT_TABLE table=default.Table_7ea27632721644359e27e0917381fb1c [id=5551c6d493d74bc39cea10e5a8198e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.410765 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.413028 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b3d1e7ce83425db0bd56457d9f2544. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:25.413251 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a88710ac9062472f80bf6fcfb82793a0 because the tablet was not found. No further retry: Not found: Tablet not found: a88710ac9062472f80bf6fcfb82793a0
I20240503 02:12:25.417047 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b91a3f3cad74f6aa6035a013cca651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.436228 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4f5be79fd35347a7ac91d7cc508b4a67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.436596 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet c823155b774e42a5a422994903eae167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:25.437708 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c823155b774e42a5a422994903eae167 because the tablet was not found. No further retry: Not found: Tablet not found: c823155b774e42a5a422994903eae167
I20240503 02:12:25.439262 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 53dad4e45a184dc1ae4a82021a48e9be with delete_type TABLET_DATA_DELETED (replaced by tablet 65774b69a8bd4664a6b55b84eba37f7f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.439674 5115 tablet_replica.cc:330] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.439879 5115 raft_consensus.cc:2229] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.440071 5115 raft_consensus.cc:2258] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.440023 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:25.440665 5115 ts_tablet_manager.cc:1875] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:25.441041 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f5be79fd35347a7ac91d7cc508b4a67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.441237 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:25.442198 5550 raft_consensus.cc:482] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.442374 5550 raft_consensus.cc:504] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.443120 5550 leader_election.cc:288] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.443290 5114 ts_tablet_manager.cc:1909] T bf9c07c7cbb743d4a8e9610281a00f94 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.443605 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 676d5f52102d496081df3964722b286f (DEFAULT_TABLE table=default.Table_681b4fa6666c4e9aaddf611003dcaa44 [id=942a3d66711342d6ac832486f109644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.444088 5547 raft_consensus.cc:482] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.444149 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676d5f52102d496081df3964722b286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.444727 5547 raft_consensus.cc:504] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.444986 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 676d5f52102d496081df3964722b286f (DEFAULT_TABLE table=default.Table_681b4fa6666c4e9aaddf611003dcaa44 [id=942a3d66711342d6ac832486f109644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.445330 5547 leader_election.cc:288] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.445364 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676d5f52102d496081df3964722b286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.447257 4811 raft_consensus.cc:1261] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:25.447301 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.447443 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1210a1217247b9801226055e2f2195 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.447573 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.445775 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.450135 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.450278 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet a88710ac9062472f80bf6fcfb82793a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.452113 4527 raft_consensus.cc:2454] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:25.452385 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 676d5f52102d496081df3964722b286f (DEFAULT_TABLE table=default.Table_681b4fa6666c4e9aaddf611003dcaa44 [id=942a3d66711342d6ac832486f109644a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.452590 4496 tablet_service.cc:1467] Processing CreateTablet for tablet ed4e84a639b2403082b737d073f00ad7 (DEFAULT_TABLE table=default.Table_6616d554610d429190adcee205cf086c [id=3cb9bed8bee34911aab76b39eaa68dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.452956 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4e84a639b2403082b737d073f00ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.453001 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 818788a24cd1419980c0d622fb05683b (DEFAULT_TABLE table=default.Table_7e6b24685afa41e388162e59492f00b0 [id=ec36effc4a6d4656b250091b3e482f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.453413 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818788a24cd1419980c0d622fb05683b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.453776 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.453979 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f (DEFAULT_TABLE table=default.Table_7ea27632721644359e27e0917381fb1c [id=5551c6d493d74bc39cea10e5a8198e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.454313 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b91a3f3cad74f6aa6035a013cca651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.454825 4672 raft_consensus.cc:1261] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:25.455169 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 (DEFAULT_TABLE table=default.Table_4a4f248f67ce475a8a517266b83052ac [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.455544 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b3d1e7ce83425db0bd56457d9f2544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.456528 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 818788a24cd1419980c0d622fb05683b (DEFAULT_TABLE table=default.Table_7e6b24685afa41e388162e59492f00b0 [id=ec36effc4a6d4656b250091b3e482f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.456845 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818788a24cd1419980c0d622fb05683b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.456987 4501 tablet_service.cc:1467] Processing CreateTablet for tablet bca69a5b48754602b04481fb6a94864f (DEFAULT_TABLE table=default.Table_1b6529a1250745eea5d10a3f304a28e7 [id=2328ebc7628741a3b60122d3669d6245]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.457271 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca69a5b48754602b04481fb6a94864f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.457516 5105 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.457695 5105 ts_tablet_manager.cc:1367] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.003s
I20240503 02:12:25.458464 5105 raft_consensus.cc:348] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.458698 5105 raft_consensus.cc:374] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.458774 5105 raft_consensus.cc:729] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.459116 5114 tablet_replica.cc:330] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.459348 5114 raft_consensus.cc:2229] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.460271 5114 raft_consensus.cc:2258] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.460597 5479 raft_consensus.cc:482] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.460753 5479 raft_consensus.cc:504] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.460937 5114 ts_tablet_manager.cc:1875] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.461046 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b4d7bc3f9a7742dfa02441ebcb165e7b" } new_table_name: "default.Table_be851867e140416c8921098eaa348201" modify_external_catalogs: true
I20240503 02:12:25.462538 5479 leader_election.cc:288] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.462493 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.462705 4811 raft_consensus.cc:2454] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:25.463407 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.464035 5105 consensus_queue.cc:260] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.464169 5547 raft_consensus.cc:482] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.464236 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet e05d88a5c49b43bea6d46feccaaa6920 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.465250 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 88b173d78ba74ede801b12cccbeef6b1 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.467378 5331 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.467561 5331 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.468219 5331 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.469947 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:25.470655 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676d5f52102d496081df3964722b286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.470832 5105 ts_tablet_manager.cc:1398] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 02:12:25.472826 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 905ca3be01e344d38350a58ac9f0d321 (DEFAULT_TABLE table=default.Table_d47e1a80afa040948b01f4ba1d3bf10b [id=74f1b122b8b3437b86a055dbb1222030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.473248 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 905ca3be01e344d38350a58ac9f0d321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.473773 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.474088 4608 leader_election.cc:302] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:25.474247 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.474697 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.471525 5547 raft_consensus.cc:504] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:25.476363 4746 leader_election.cc:341] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:25.476785 4746 leader_election.cc:341] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.476905 4746 leader_election.cc:302] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:25.477506 5105 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:25.477564 5331 raft_consensus.cc:2726] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.479351 5105 tablet_bootstrap.cc:654] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:25.479497 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1436dd3df7284f9bb379236eb9d76836 consensus state has no leader
I20240503 02:12:25.483229 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet bf9c07c7cbb743d4a8e9610281a00f94 (table default.Table_21c620bc30de4de5b37ce3993ff24d1a [id=bb6040194756494b930862f84ed0f263]) successfully deleted
I20240503 02:12:25.485440 5103 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:25.485597 5103 ts_tablet_manager.cc:1367] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.247s user 0.035s sys 0.117s
I20240503 02:12:25.486411 5103 raft_consensus.cc:348] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.486614 5103 raft_consensus.cc:374] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.486698 5103 raft_consensus.cc:729] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.486889 5103 consensus_queue.cc:260] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.487473 5103 ts_tablet_manager.cc:1398] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:25.487787 5103 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:25.489462 5479 raft_consensus.cc:2781] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.489585 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 818788a24cd1419980c0d622fb05683b (DEFAULT_TABLE table=default.Table_7e6b24685afa41e388162e59492f00b0 [id=ec36effc4a6d4656b250091b3e482f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.490000 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818788a24cd1419980c0d622fb05683b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.488637 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.490595 5479 raft_consensus.cc:482] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.494864 5547 leader_election.cc:288] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.495548 5479 raft_consensus.cc:3037] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.495520 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:25.498327 5479 raft_consensus.cc:504] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.498993 5479 leader_election.cc:288] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:25.501024 5115 ts_tablet_manager.cc:1888] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.501143 5115 log.cc:1196] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/53dad4e45a184dc1ae4a82021a48e9be
I20240503 02:12:25.501556 5115 ts_tablet_manager.cc:1909] T 53dad4e45a184dc1ae4a82021a48e9be P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:25.501664 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e94031465ce7407dab0f90ad8809ac21 (table default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet e94031465ce7407dab0f90ad8809ac21 consensus state has no leader
I20240503 02:12:25.502735 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:25.502969 4811 raft_consensus.cc:3037] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.505843 4811 raft_consensus.cc:2454] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:25.510346 4608 leader_election.cc:302] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:25.510852 5479 raft_consensus.cc:2781] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.511260 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4101d1a1a4c14c018fecdd3b3d05641f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.511360 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b8df46519f1a401b862bbfabff3577fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.511843 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4101d1a1a4c14c018fecdd3b3d05641f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.511893 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8df46519f1a401b862bbfabff3577fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.513192 5479 raft_consensus.cc:686] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:25.513526 5479 consensus_queue.cc:237] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:25.517408 4746 leader_election.cc:341] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.517560 4746 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:25.518169 5331 raft_consensus.cc:2726] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.519527 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:25.522532 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.524215 5103 tablet_bootstrap.cc:654] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.530733 5106 tablet_bootstrap.cc:492] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.530925 5106 ts_tablet_manager.cc:1367] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.172s user 0.011s sys 0.000s
I20240503 02:12:25.531849 5106 raft_consensus.cc:348] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.532112 5106 raft_consensus.cc:374] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.532212 5106 raft_consensus.cc:729] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.532452 5106 consensus_queue.cc:260] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.533150 5106 ts_tablet_manager.cc:1398] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:25.533469 5106 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.535127 5106 tablet_bootstrap.cc:654] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.529958 5159 ts_tablet_manager.cc:1888] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.536268 4648 tablet_service.cc:1467] Processing CreateTablet for tablet a88710ac9062472f80bf6fcfb82793a0 (DEFAULT_TABLE table=default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.536692 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88710ac9062472f80bf6fcfb82793a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.537940 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.542193 4503 tablet_service.cc:1467] Processing CreateTablet for tablet a88710ac9062472f80bf6fcfb82793a0 (DEFAULT_TABLE table=default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.542717 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88710ac9062472f80bf6fcfb82793a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.547355 5159 log.cc:1196] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:25.547857 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0ea8c89cbb9e45d9af49c75e9921666e" } modify_external_catalogs: true
I20240503 02:12:25.548110 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0ea8c89cbb9e45d9af49c75e9921666e" } modify_external_catalogs: true
W20240503 02:12:25.549036 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e366ce9d79b3458fb26e96b6e9278fbe consensus state has no leader
I20240503 02:12:25.549474 4781 tablet_service.cc:1467] Processing CreateTablet for tablet a88710ac9062472f80bf6fcfb82793a0 (DEFAULT_TABLE table=default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.549851 5159 ts_tablet_manager.cc:1909] T c0c18cecefab4b6caed0039d09cc1890 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:25.549921 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88710ac9062472f80bf6fcfb82793a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.550747 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 329) took 2307 ms. Trace:
I20240503 02:12:25.550885 5159 rpcz_store.cc:276] 0503 02:12:23.243042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.245463 (+ 2421us) service_pool.cc:225] Handling call
0503 02:12:25.325055 (+2079592us) ts_tablet_manager.cc:1120] Deleting tablet c0c18cecefab4b6caed0039d09cc1890
0503 02:12:25.325065 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.443716 (+118651us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.488362 (+ 44646us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:25.550698 (+ 62336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063595}
I20240503 02:12:25.551144 4275 catalog_manager.cc:5484] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.551631 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
I20240503 02:12:25.551898 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:25.552101 4275 catalog_manager.cc:5484] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.552597 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:25.553248 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d689749a9c6a4c91a19f092325e08e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.553879 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d689749a9c6a4c91a19f092325e08e73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.556525 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 342) took 2033 ms. Trace:
I20240503 02:12:25.556655 5159 rpcz_store.cc:276] 0503 02:12:23.522904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.532694 (+ 9790us) service_pool.cc:225] Handling call
0503 02:12:25.555876 (+2023182us) ts_tablet_manager.cc:1120] Deleting tablet b38d63d2cc8c4f54ba43cdf8cde1841f
0503 02:12:25.555889 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.556486 (+ 597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017777}
I20240503 02:12:25.557127 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_140dd34e051a4448bfeeafb111265836" } new_table_name: "default.Table_de40cf71b0974c799cf87de718e159b4" modify_external_catalogs: true
W20240503 02:12:25.561966 4746 consensus_peers.cc:485] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:25.562418 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1975c790bfa4449baa6d7b81509c5003 consensus state has no leader
W20240503 02:12:25.563640 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42e80cc31f5c4908b82f2b030255f6af consensus state has no leader
I20240503 02:12:25.564028 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.564545 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 343) took 2041 ms. Trace:
I20240503 02:12:25.564664 5159 rpcz_store.cc:276] 0503 02:12:23.530712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.531279 (+ 567us) service_pool.cc:225] Handling call
0503 02:12:25.564370 (+2033091us) ts_tablet_manager.cc:1120] Deleting tablet 3b41dce88919456aa846dbdcf5e2759f
0503 02:12:25.564384 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.564507 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019498}
I20240503 02:12:25.566184 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:25.566447 4528 raft_consensus.cc:3037] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.567410 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.568017 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.568265 4527 raft_consensus.cc:2454] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:25.568784 5114 ts_tablet_manager.cc:1888] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.570694 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 345) took 2014 ms. Trace:
I20240503 02:12:25.571977 5159 rpcz_store.cc:276] 0503 02:12:23.556501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.556676 (+ 175us) service_pool.cc:225] Handling call
0503 02:12:25.570501 (+2013825us) ts_tablet_manager.cc:1120] Deleting tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
0503 02:12:25.570515 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.570655 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1978075}
I20240503 02:12:25.573415 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 346) took 2010 ms. Trace:
I20240503 02:12:25.574644 5159 rpcz_store.cc:276] 0503 02:12:23.563208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.592464 (+ 29256us) service_pool.cc:225] Handling call
0503 02:12:25.573248 (+1980784us) ts_tablet_manager.cc:1120] Deleting tablet b161628065704c798020edbe098844c2
0503 02:12:25.573261 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.573380 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963969}
I20240503 02:12:25.575945 5106 tablet_bootstrap.cc:492] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.576656 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 348) took 2003 ms. Trace:
I20240503 02:12:25.577181 5106 ts_tablet_manager.cc:1367] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.008s
I20240503 02:12:25.577334 5159 rpcz_store.cc:276] 0503 02:12:23.573071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.612291 (+ 39220us) service_pool.cc:225] Handling call
0503 02:12:25.576501 (+1964210us) ts_tablet_manager.cc:1120] Deleting tablet 41d44fe8f6334572996665bdec60e2e5
0503 02:12:25.576515 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.576624 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963814}
I20240503 02:12:25.569029 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:25.577988 4527 raft_consensus.cc:3037] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.578496 5106 raft_consensus.cc:348] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.578734 5106 raft_consensus.cc:374] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.578826 5106 raft_consensus.cc:729] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.579355 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 347) took 2016 ms. Trace:
I20240503 02:12:25.579468 5159 rpcz_store.cc:276] 0503 02:12:23.563326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.609319 (+ 45993us) service_pool.cc:225] Handling call
0503 02:12:25.579207 (+1969888us) ts_tablet_manager.cc:1120] Deleting tablet 2f862891b43a4845b723bfdb9a7da591
0503 02:12:25.579220 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.579321 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940798}
I20240503 02:12:25.580058 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 356) took 1709 ms. Trace:
I20240503 02:12:25.580179 5159 rpcz_store.cc:276] 0503 02:12:23.870846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:23.872386 (+ 1540us) service_pool.cc:225] Handling call
0503 02:12:25.579882 (+1707496us) ts_tablet_manager.cc:1120] Deleting tablet b84856917cb84c2a853c9a8a80246704
0503 02:12:25.579893 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.580025 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640595}
I20240503 02:12:25.580619 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 366) took 1101 ms. Trace:
I20240503 02:12:25.580730 5159 rpcz_store.cc:276] 0503 02:12:24.478919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.479310 (+ 391us) service_pool.cc:225] Handling call
0503 02:12:25.580507 (+1101197us) ts_tablet_manager.cc:1120] Deleting tablet 27c13f257eae44689af07cc31ea254c4
0503 02:12:25.580515 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.580587 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1082539}
I20240503 02:12:25.581306 5114 log.cc:1196] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/2a51563396614c54b980f0679e84a2b1
I20240503 02:12:25.581214 5106 consensus_queue.cc:260] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.581868 5114 ts_tablet_manager.cc:1909] T 2a51563396614c54b980f0679e84a2b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.582669 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 368) took 1086 ms. Trace:
I20240503 02:12:25.583235 5114 tablet_replica.cc:330] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.583460 5114 raft_consensus.cc:2229] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.583626 5114 raft_consensus.cc:2258] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.584822 4673 raft_consensus.cc:3037] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.585247 4527 raft_consensus.cc:2454] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:25.576345 4528 raft_consensus.cc:2454] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:25.585702 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:25.585956 4529 raft_consensus.cc:2454] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:25.586532 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959d998ec4ec4a52a6031650f2da4691" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:25.586756 4529 raft_consensus.cc:3037] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.588186 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 53dad4e45a184dc1ae4a82021a48e9be (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
I20240503 02:12:25.588218 5426 consensus_queue.cc:1035] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.589120 4462 leader_election.cc:302] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:25.589404 4462 raft_consensus.cc:2670] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:25.590983 4529 raft_consensus.cc:2454] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:25.597069 5547 consensus_queue.cc:1035] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.597433 5114 ts_tablet_manager.cc:1875] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:25.598290 4462 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:25.598667 5547 consensus_queue.cc:1035] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.598850 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_95c5dbfc7e2542dd93ea5839f66200e1" } modify_external_catalogs: true
W20240503 02:12:25.599311 4462 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 7d1210a1217247b9801226055e2f2195
I20240503 02:12:25.599604 5547 consensus_queue.cc:1035] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:25.600782 4462 leader_election.cc:341] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 6cd402c0eb9e4cf49dc722f2dd214e09
I20240503 02:12:25.601023 4462 leader_election.cc:302] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:25.601256 4462 raft_consensus.cc:2670] T 6cd402c0eb9e4cf49dc722f2dd214e09 P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:25.599073 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_95c5dbfc7e2542dd93ea5839f66200e1" } modify_external_catalogs: true
I20240503 02:12:25.600098 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23083e3bb77444ec82de47917ee95f01" } modify_external_catalogs: true
I20240503 02:12:25.602954 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23083e3bb77444ec82de47917ee95f01" } modify_external_catalogs: true
I20240503 02:12:25.600270 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.603756 5103 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
W20240503 02:12:25.603750 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.603917 5103 ts_tablet_manager.cc:1367] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.000s
I20240503 02:12:25.604502 5547 consensus_queue.cc:1035] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.600742 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1d1adcf60a414b9b8d0d7621be5c21c8" } modify_external_catalogs: true
I20240503 02:12:25.605182 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1d1adcf60a414b9b8d0d7621be5c21c8" } modify_external_catalogs: true
I20240503 02:12:25.605201 5103 raft_consensus.cc:348] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.605417 5103 raft_consensus.cc:374] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.605487 5103 raft_consensus.cc:729] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.605763 5103 consensus_queue.cc:260] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.606804 5103 ts_tablet_manager.cc:1398] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 02:12:25.610257 4462 leader_election.cc:341] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 7d1210a1217247b9801226055e2f2195
I20240503 02:12:25.610375 4462 leader_election.cc:302] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:25.613729 5114 ts_tablet_manager.cc:1888] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:25.616791 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.617110 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.617543 5114 log.cc:1196] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/53dad4e45a184dc1ae4a82021a48e9be
I20240503 02:12:25.617522 4608 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:25.617992 5114 ts_tablet_manager.cc:1909] T 53dad4e45a184dc1ae4a82021a48e9be P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.617959 4608 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:25.614445 5103 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:25.614590 5547 raft_consensus.cc:2726] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.618659 5382 raft_consensus.cc:2726] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.616120 5550 consensus_queue.cc:1035] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.618824 5382 raft_consensus.cc:2726] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.619448 4575 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:25.621563 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.621814 4674 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:25.624190 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b9003f248b47a9952e6f995887ae0c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.624503 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.624632 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.624713 4813 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:25.624831 4810 raft_consensus.cc:2379] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:25.624966 4811 raft_consensus.cc:2379] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:25.629315 4462 leader_election.cc:302] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:25.629802 5550 raft_consensus.cc:2781] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.629911 5550 raft_consensus.cc:482] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.630021 5550 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.632112 4859 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:25.633834 5550 raft_consensus.cc:504] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.634799 5550 leader_election.cc:288] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.635636 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:25.635895 4674 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.636059 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:25.636169 4462 leader_election.cc:302] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:25.636296 4810 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.636416 5550 raft_consensus.cc:2726] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.636821 4462 leader_election.cc:302] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:25.637064 5550 raft_consensus.cc:2726] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.639891 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e8a9dc27e1654f2e92a1bd71999f7959" } modify_external_catalogs: true
I20240503 02:12:25.640228 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e8a9dc27e1654f2e92a1bd71999f7959" } modify_external_catalogs: true
I20240503 02:12:25.641126 4674 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:25.641697 4462 leader_election.cc:302] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:25.642050 5550 raft_consensus.cc:2781] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.642225 5550 raft_consensus.cc:686] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:25.642481 5550 consensus_queue.cc:237] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.643044 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 94559cfc32704eae96b121f165686edc
W20240503 02:12:25.647163 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b92999890ee49d78d7c16f5ed1c4958 has no consensus state
I20240503 02:12:25.649358 5103 tablet_bootstrap.cc:654] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.650542 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 79b8db679ba24ac6a93be591c0513ab0
I20240503 02:12:25.652550 5114 tablet_replica.cc:330] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.652619 4810 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:25.652807 5114 raft_consensus.cc:2229] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.653050 5114 raft_consensus.cc:2258] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.653880 5114 ts_tablet_manager.cc:1875] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.654922 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ea5e4db5204e4acf8719100d02d533fb
I20240503 02:12:25.656090 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 46560ac6d94943d2a8fb3b3904f2b5b6
W20240503 02:12:25.657974 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 59734.4us
I20240503 02:12:25.660516 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.661140 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.663154 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 2a51563396614c54b980f0679e84a2b1 (table default.Table_5e9ba5798f4c4414826196e0761c2df9 [id=0a6dd433bee8497395826808bba7774e]) successfully deleted
I20240503 02:12:25.668354 5550 raft_consensus.cc:482] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.671190 5550 raft_consensus.cc:504] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.671901 5550 leader_election.cc:288] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.675272 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.675801 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.676301 4673 raft_consensus.cc:2454] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:25.676605 4810 raft_consensus.cc:2454] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:25.677289 4462 leader_election.cc:302] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:25.677736 5550 raft_consensus.cc:2781] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.677843 5550 raft_consensus.cc:482] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.677911 5550 raft_consensus.cc:3037] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.670993 5103 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:25.679996 5103 ts_tablet_manager.cc:1367] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.000s
I20240503 02:12:25.680526 5550 raft_consensus.cc:504] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.681380 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3f5c5ee57b8c4f27987de424a90281cb" } new_table_name: "default.Table_4f9e55262006461d8fd7dd706e640469" modify_external_catalogs: true
I20240503 02:12:25.683002 5550 leader_election.cc:288] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.683305 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:25.683574 4811 raft_consensus.cc:3037] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.685037 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:25.685304 4674 raft_consensus.cc:3037] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.686869 4811 raft_consensus.cc:2454] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:25.687552 4462 leader_election.cc:302] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:25.687886 5550 raft_consensus.cc:2781] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.688066 5550 raft_consensus.cc:686] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:25.688409 5550 consensus_queue.cc:237] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.688671 4674 raft_consensus.cc:2454] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
W20240503 02:12:25.689178 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 24737f8a8ca74996b9f54d49d12b9288 because the tablet was not found. No further retry: Not found: Tablet not found: 24737f8a8ca74996b9f54d49d12b9288
I20240503 02:12:25.693318 5103 raft_consensus.cc:348] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.694350 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 53dad4e45a184dc1ae4a82021a48e9be (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
W20240503 02:12:25.695219 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet aa699a1dc3c34be59925560d5e11f49e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa699a1dc3c34be59925560d5e11f49e already in progress: creating tablet
W20240503 02:12:25.695979 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8a90ebcf29004d1a925b69fa20998dd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a90ebcf29004d1a925b69fa20998dd3 already in progress: creating tablet
W20240503 02:12:25.696611 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ff8d2c0991e849b5afa35e7230db298d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8d2c0991e849b5afa35e7230db298d already in progress: creating tablet
I20240503 02:12:25.697108 5103 raft_consensus.cc:374] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.697584 5103 raft_consensus.cc:729] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:25.697520 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4b4972057fae45168756d7c28ddbe2bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4972057fae45168756d7c28ddbe2bf already in progress: creating tablet
I20240503 02:12:25.701445 5103 consensus_queue.cc:260] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.706097 5103 ts_tablet_manager.cc:1398] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:12:25.706321 5590 tablet.cc:1722] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:25.706485 5103 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:25.708761 5103 tablet_bootstrap.cc:654] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.709861 5591 tablet.cc:1722] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:25.703786 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ae1fa81b7e334c958fc7d781ecd7bc11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.715976 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae1fa81b7e334c958fc7d781ecd7bc11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.716603 5114 ts_tablet_manager.cc:1888] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:25.698261 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5dbbfa2d1a49400c809d5b476036cdcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dbbfa2d1a49400c809d5b476036cdcd already in progress: creating tablet
I20240503 02:12:25.723040 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7dbf326ce1e445b2a3d9f194413f2196"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.723649 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dbf326ce1e445b2a3d9f194413f2196 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.725726 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 3db94fb9d53044d796e2973143d888fb from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.728277 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a0e1c2fa9cb3481dafb9d7f1e826a30c
I20240503 02:12:25.731096 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_466756ec40864212abe6bd1a32875b14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.731673 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_466756ec40864212abe6bd1a32875b14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.733991 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 1436dd3df7284f9bb379236eb9d76836 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.735970 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8b6599206549dd8ddf449a52c61fce (DEFAULT_TABLE table=default.Table_8cdd736fadb64d9faf9eb743916fa0b8 [id=c11dd6cb733044ef94c59992e98601f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.736516 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8b6599206549dd8ddf449a52c61fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.748337 5114 log.cc:1196] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7d1210a1217247b9801226055e2f2195
I20240503 02:12:25.751060 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3db94fb9d53044d796e2973143d888fb
I20240503 02:12:25.751725 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 3db94fb9d53044d796e2973143d888fb deleted and replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0
I20240503 02:12:25.759599 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fd7702f08f474fd5beb82be6891ac6f8" } new_table_name: "default.Table_0e5ff92986ff447dbc2c9e69a628e8fe" modify_external_catalogs: true
W20240503 02:12:25.760396 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7461aa76a1af41cb9253285bda708f30 consensus state has no leader
I20240503 02:12:25.771598 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e530bacca394443cbd9883735523abf1" } new_table_name: "default.Table_873e48ca8c434f55927075d87ac065b1" modify_external_catalogs: true
I20240503 02:12:25.773155 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1e24815702a84aa281e889334d8b734f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.787256 4792 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.787910 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2850c07d042a7951a5fbeb0e4945d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.797174 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1c7b1c385fa46b0a311b8957066d4eb" } new_table_name: "default.Table_7fe2fd4286f4411aad562211e4bd77ae" modify_external_catalogs: true
I20240503 02:12:25.797341 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_47b984d6c813457786ac8b9c4fad0255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.798774 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47b984d6c813457786ac8b9c4fad0255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.800671 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e24815702a84aa281e889334d8b734f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.802649 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dedead5a2bf347f6991cce8e9b4db708 consensus state has no leader
I20240503 02:12:25.808799 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.809398 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.809777 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:25.810709 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:25.819461 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607e49cde2fd4aab9f44921d3c3921a5 consensus state has no leader
W20240503 02:12:25.823691 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 205192us
I20240503 02:12:25.828122 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8b6599206549dd8ddf449a52c61fce (DEFAULT_TABLE table=default.Table_8cdd736fadb64d9faf9eb743916fa0b8 [id=c11dd6cb733044ef94c59992e98601f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.828639 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8b6599206549dd8ddf449a52c61fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.854362 5105 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.854813 5105 ts_tablet_manager.cc:1367] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.378s user 0.007s sys 0.006s
I20240503 02:12:25.856328 5105 raft_consensus.cc:348] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.856741 5105 raft_consensus.cc:374] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.856976 5105 raft_consensus.cc:729] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.857466 5105 consensus_queue.cc:260] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.583396 5159 rpcz_store.cc:276] 0503 02:12:24.495895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.505019 (+ 9124us) service_pool.cc:225] Handling call
0503 02:12:25.582532 (+1077513us) ts_tablet_manager.cc:1120] Deleting tablet aa67857b858c4719b498291c52b7aceb
0503 02:12:25.582542 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.582634 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077198}
I20240503 02:12:25.859261 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 367) took 1373 ms. Trace:
I20240503 02:12:25.859537 5106 ts_tablet_manager.cc:1398] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:25.863113 5106 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.865842 5106 tablet_bootstrap.cc:654] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.869904 4648 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.870440 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2850c07d042a7951a5fbeb0e4945d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.873346 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.873834 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.873960 5550 raft_consensus.cc:482] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.876574 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.876637 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.877235 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.874953 5550 raft_consensus.cc:504] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.877471 5115 tablet_replica.cc:330] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.877681 5115 raft_consensus.cc:2229] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.877856 5115 raft_consensus.cc:2258] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.878741 5115 ts_tablet_manager.cc:1875] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.883860 5105 ts_tablet_manager.cc:1398] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:25.884276 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:25.884498 5159 rpcz_store.cc:276] 0503 02:12:24.485754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.486165 (+ 411us) service_pool.cc:225] Handling call
0503 02:12:25.859097 (+1372932us) ts_tablet_manager.cc:1120] Deleting tablet 34eb69d5f4bd427a8e430a00fb076f83
0503 02:12:25.859110 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.859230 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1353653}
I20240503 02:12:25.884959 5115 ts_tablet_manager.cc:1888] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:25.885144 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 369) took 1318 ms. Trace:
I20240503 02:12:25.885265 5159 rpcz_store.cc:276] 0503 02:12:24.566820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.568982 (+ 2162us) service_pool.cc:225] Handling call
0503 02:12:25.884968 (+1315986us) ts_tablet_manager.cc:1120] Deleting tablet 87713e899d2148308316d807a6d76921
0503 02:12:25.884980 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.885110 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315575}
I20240503 02:12:25.885787 5115 log.cc:1196] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/2a51563396614c54b980f0679e84a2b1
I20240503 02:12:25.885778 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 370) took 1257 ms. Trace:
I20240503 02:12:25.885885 5159 rpcz_store.cc:276] 0503 02:12:24.628539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.628607 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:25.885646 (+1257039us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:25.885654 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.885736 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248519}
I20240503 02:12:25.886240 5115 ts_tablet_manager.cc:1909] T 2a51563396614c54b980f0679e84a2b1 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:25.886387 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 371) took 1238 ms. Trace:
I20240503 02:12:25.886493 5159 rpcz_store.cc:276] 0503 02:12:24.648211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.648268 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:25.886256 (+1237988us) ts_tablet_manager.cc:1120] Deleting tablet 9ed525a61c734ee2b23d5d4f319a7d67
0503 02:12:25.886265 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.886347 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237749}
I20240503 02:12:25.886970 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 372) took 1187 ms. Trace:
I20240503 02:12:25.887154 5159 rpcz_store.cc:276] 0503 02:12:24.699377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.699443 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:25.886798 (+1187355us) ts_tablet_manager.cc:1120] Deleting tablet d607da9689c14e51add9178a48b35913
0503 02:12:25.886806 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.886886 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187072}
I20240503 02:12:25.887907 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.888198 5115 tablet_replica.cc:330] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:25.888432 5115 raft_consensus.cc:2229] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:25.888727 5115 pending_rounds.cc:62] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:25.889008 5115 pending_rounds.cc:68] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420807226867712 op_type: NO_OP noop_request { }
I20240503 02:12:25.889250 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.889142 5159 tablet_replica.cc:330] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:25.889618 5115 raft_consensus.cc:2866] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:25.889732 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:25.889839 5159 raft_consensus.cc:2229] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:25.889986 4812 raft_consensus.cc:2454] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:25.889816 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.890239 5115 raft_consensus.cc:2258] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.890430 5159 raft_consensus.cc:2258] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.890429 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:25.891808 5105 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:25.892300 5382 consensus_queue.cc:1035] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.892648 4509 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.893934 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2850c07d042a7951a5fbeb0e4945d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.896412 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8b6599206549dd8ddf449a52c61fce (DEFAULT_TABLE table=default.Table_8cdd736fadb64d9faf9eb743916fa0b8 [id=c11dd6cb733044ef94c59992e98601f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.896859 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8b6599206549dd8ddf449a52c61fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.898470 5115 ts_tablet_manager.cc:1875] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.901511 5159 ts_tablet_manager.cc:1875] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.903093 5550 leader_election.cc:288] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.905714 5105 tablet_bootstrap.cc:654] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.906605 4462 leader_election.cc:302] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:25.907461 5550 raft_consensus.cc:2781] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.907582 5550 raft_consensus.cc:482] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.909291 5550 raft_consensus.cc:3037] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.918390 5106 tablet_bootstrap.cc:492] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.918679 5106 ts_tablet_manager.cc:1367] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240503 02:12:25.920748 5550 raft_consensus.cc:504] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.921938 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:25.923017 5106 raft_consensus.cc:348] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.923359 5106 raft_consensus.cc:374] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.923499 5106 raft_consensus.cc:729] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.923796 5106 consensus_queue.cc:260] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.924470 5106 ts_tablet_manager.cc:1398] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:25.924829 5106 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:25.926618 5106 tablet_bootstrap.cc:654] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.926724 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4489651bcc554f9ca32e9f7a9629b9ce" } modify_external_catalogs: true
I20240503 02:12:25.927742 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_29fbbe157bc0404fa71f43052d345a47" } new_table_name: "default.Table_c7da59f96b8141f093a1b8de9a334e5d" modify_external_catalogs: true
I20240503 02:12:25.928367 5550 leader_election.cc:288] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:25.930434 4675 raft_consensus.cc:3037] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.933674 4675 raft_consensus.cc:2454] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:25.933914 4721 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:25.939354 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:25.941880 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c0c18cecefab4b6caed0039d09cc1890 (table default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]) successfully deleted
I20240503 02:12:25.942483 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3d65228f6c2b4694bb60f3118ad65af7" } modify_external_catalogs: true
I20240503 02:12:25.942672 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3d65228f6c2b4694bb60f3118ad65af7" } modify_external_catalogs: true
I20240503 02:12:25.943078 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a5a36396c8974489a65a6510c520cee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.943663 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5a36396c8974489a65a6510c520cee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.944973 5114 ts_tablet_manager.cc:1909] T 7d1210a1217247b9801226055e2f2195 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:25.941911 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4489651bcc554f9ca32e9f7a9629b9ce" } modify_external_catalogs: true
I20240503 02:12:25.945914 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c4de876e162a48c1bba5469a0c9d6a34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.947197 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4bb66ff27eed4eb9b2513d844a9bdbbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.947628 5114 tablet_replica.cc:330] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:25.947850 5114 raft_consensus.cc:2229] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.947937 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d47e1a80afa040948b01f4ba1d3bf10b" } modify_external_catalogs: true
I20240503 02:12:25.948019 5114 raft_consensus.cc:2258] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:25.948134 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1436dd3df7284f9bb379236eb9d76836 consensus state has no leader
I20240503 02:12:25.948581 5114 ts_tablet_manager.cc:1875] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:25.938546 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bb9ac773c4bc685cb8193e574460b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:25.949479 4811 raft_consensus.cc:3037] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.950623 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_df799d52e6434e0f90b831960d0f39c1" } modify_external_catalogs: true
I20240503 02:12:25.952021 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_df799d52e6434e0f90b831960d0f39c1" } modify_external_catalogs: true
I20240503 02:12:25.952467 4811 raft_consensus.cc:2454] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:25.952448 4462 leader_election.cc:302] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:25.953140 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4de876e162a48c1bba5469a0c9d6a34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.954865 5105 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:25.955067 5105 ts_tablet_manager.cc:1367] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.071s user 0.000s sys 0.005s
I20240503 02:12:25.956312 5105 raft_consensus.cc:348] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.956645 5105 raft_consensus.cc:374] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.956840 5105 raft_consensus.cc:729] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.957365 5105 consensus_queue.cc:260] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.957650 5589 raft_consensus.cc:2781] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.957794 5589 raft_consensus.cc:686] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:25.958122 5589 consensus_queue.cc:237] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:25.959080 5105 ts_tablet_manager.cc:1398] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:25.959681 5105 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:25.960696 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c2a8190210439fb7589cb02ec82bc2 consensus state has no leader
I20240503 02:12:25.961015 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 4b4972057fae45168756d7c28ddbe2bf from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:25.961268 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb66ff27eed4eb9b2513d844a9bdbbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.962400 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d47e1a80afa040948b01f4ba1d3bf10b" } modify_external_catalogs: true
I20240503 02:12:25.962898 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_334ae1c85436463dad80d97f857112a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.963418 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_334ae1c85436463dad80d97f857112a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.969034 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 043ba14eaa5b46319b0b88457d2a5f45 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:25.969834 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b38d63d2cc8c4f54ba43cdf8cde1841f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38d63d2cc8c4f54ba43cdf8cde1841f already in progress: creating tablet
W20240503 02:12:25.970332 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3b41dce88919456aa846dbdcf5e2759f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b41dce88919456aa846dbdcf5e2759f already in progress: creating tablet
W20240503 02:12:25.970691 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ced4dfb2cf541af98d50d22d1a0f0ac already in progress: creating tablet
W20240503 02:12:25.970978 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b161628065704c798020edbe098844c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b161628065704c798020edbe098844c2 already in progress: creating tablet
W20240503 02:12:25.971225 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 41d44fe8f6334572996665bdec60e2e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41d44fe8f6334572996665bdec60e2e5 already in progress: creating tablet
W20240503 02:12:25.971546 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2f862891b43a4845b723bfdb9a7da591 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f862891b43a4845b723bfdb9a7da591 already in progress: creating tablet
W20240503 02:12:25.971866 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b84856917cb84c2a853c9a8a80246704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84856917cb84c2a853c9a8a80246704 already in progress: creating tablet
W20240503 02:12:25.972157 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 27c13f257eae44689af07cc31ea254c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27c13f257eae44689af07cc31ea254c4 already in progress: creating tablet
W20240503 02:12:25.972647 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet aa67857b858c4719b498291c52b7aceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67857b858c4719b498291c52b7aceb already in progress: creating tablet
I20240503 02:12:25.978961 5382 consensus_queue.cc:1035] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:25.980669 4170 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 74242.9us
I20240503 02:12:25.988907 4671 raft_consensus.cc:2362] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1. Re-sending same reply.
I20240503 02:12:25.989234 5106 tablet_bootstrap.cc:492] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:25.989953 5106 ts_tablet_manager.cc:1367] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.004s
I20240503 02:12:25.975888 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:25.983327 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 95681e67db1e4485876d530133e8a514 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:25.991365 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]). Expected version 2 got 0
I20240503 02:12:25.992131 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1de4326d5bda4d70afc8b879ab9acb6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:25.992647 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d6eeea17720743c8b78544cd9e6efefd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.993199 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6eeea17720743c8b78544cd9e6efefd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.993575 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de4326d5bda4d70afc8b879ab9acb6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:25.991492 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_98865c1dc735482d9fdb50d75f612ac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:25.994786 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98865c1dc735482d9fdb50d75f612ac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:25.995750 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 34eb69d5f4bd427a8e430a00fb076f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34eb69d5f4bd427a8e430a00fb076f83 already in progress: creating tablet
I20240503 02:12:25.984064 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_150a5f1631d94beaa4e6b81e2e9ade3d" } modify_external_catalogs: true
I20240503 02:12:25.996428 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_150a5f1631d94beaa4e6b81e2e9ade3d" } modify_external_catalogs: true
W20240503 02:12:25.995994 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 87713e899d2148308316d807a6d76921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87713e899d2148308316d807a6d76921 already in progress: creating tablet
W20240503 02:12:25.997321 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9ed525a61c734ee2b23d5d4f319a7d67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed525a61c734ee2b23d5d4f319a7d67 already in progress: creating tablet
W20240503 02:12:25.997535 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d607da9689c14e51add9178a48b35913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d607da9689c14e51add9178a48b35913 already in progress: creating tablet
W20240503 02:12:25.997742 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 24737f8a8ca74996b9f54d49d12b9288 because the tablet was not found. No further retry: Not found: Tablet not found: 24737f8a8ca74996b9f54d49d12b9288
W20240503 02:12:25.997948 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 86d719c58974442281c7d694e1ca485e because the tablet was not found. No further retry: Not found: Tablet not found: 86d719c58974442281c7d694e1ca485e
I20240503 02:12:25.998315 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7d1210a1217247b9801226055e2f2195 (table default.Table_1845b8fdfc7b488f9e5050933d9d4cdf [id=6125cf8c512f45acaf783339df17064d]) successfully deleted
I20240503 02:12:25.991544 5106 raft_consensus.cc:348] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:25.998525 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 43119471f91c4a1ba900f267c88db9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43119471f91c4a1ba900f267c88db9a6 already in progress: creating tablet
I20240503 02:12:25.998625 5106 raft_consensus.cc:374] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:25.998716 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 84b6e303e34a465caffed6c9e1e6bf74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b6e303e34a465caffed6c9e1e6bf74 already in progress: creating tablet
W20240503 02:12:25.998904 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a88710ac9062472f80bf6fcfb82793a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88710ac9062472f80bf6fcfb82793a0 already in progress: creating tablet
W20240503 02:12:25.999130 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8d0494ee4bd746069bfaf39bfb46687d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d0494ee4bd746069bfaf39bfb46687d already in progress: creating tablet
W20240503 02:12:25.999351 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 76b01907833d44f6a000a4de3a2a6e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76b01907833d44f6a000a4de3a2a6e96 already in progress: creating tablet
I20240503 02:12:26.000110 5106 raft_consensus.cc:729] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.000607 5103 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.000851 5103 ts_tablet_manager.cc:1367] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.294s user 0.062s sys 0.103s
I20240503 02:12:26.001160 5106 consensus_queue.cc:260] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.001960 5106 ts_tablet_manager.cc:1398] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:12:26.002616 5106 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:26.005079 5105 tablet_bootstrap.cc:654] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.009573 5577 consensus_queue.cc:1035] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:26.009331 5602 tablet.cc:1722] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:26.012578 5103 raft_consensus.cc:348] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.014827 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 905ca3be01e344d38350a58ac9f0d321
I20240503 02:12:26.020637 5103 raft_consensus.cc:374] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.020761 5103 raft_consensus.cc:729] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.021049 5103 consensus_queue.cc:260] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.022639 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1436dd3df7284f9bb379236eb9d76836
I20240503 02:12:26.022694 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.023025 4274 catalog_manager.cc:6554] ReplaceTablet: tablet 1436dd3df7284f9bb379236eb9d76836 deleted and replaced by tablet 17a24b5c83b649e79644b08253d851d6
I20240503 02:12:26.023803 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.023873 5103 ts_tablet_manager.cc:1398] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:12:26.023968 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c2a0d32aa38141e8bd72a2e2293843c2" } new_table_name: "default.Table_b839b2ab5a8a4a399e63762bec827155" modify_external_catalogs: true
I20240503 02:12:26.025288 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3aad7d6dcf2e4009872ce156c536f38e
I20240503 02:12:26.026091 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.026629 5103 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.026628 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 043ba14eaa5b46319b0b88457d2a5f45
I20240503 02:12:26.026907 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3df92b6e544de9ca76d98706eebec (DEFAULT_TABLE table=default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8 [id=d636c15b30004d098fa523ef493335db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.026985 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet cd66f88a24b74f2c9b2050af081d4601
I20240503 02:12:26.027335 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3df92b6e544de9ca76d98706eebec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.028481 5103 tablet_bootstrap.cc:654] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.028434 5159 ts_tablet_manager.cc:1888] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:26.030083 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8eb9b75d260a42229a49d6343e0b2e3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.030184 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 76aef6b021da4f5aad7f523b15d484ec (DEFAULT_TABLE table=default.Table_4f5be79fd35347a7ac91d7cc508b4a67 [id=b078c65aa39343aabe02f29bf833abe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.030575 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aef6b021da4f5aad7f523b15d484ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:26.030645 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eb9b75d260a42229a49d6343e0b2e3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:26.031944 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 28c1a81589df496b92b980353f00e594 (DEFAULT_TABLE table=default.Table_6d070a69982c45b7904a12c6df0386b6 [id=6f362fd85655487d93025664e6a68fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.032330 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c1a81589df496b92b980353f00e594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.033696 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 (DEFAULT_TABLE table=default.Table_a2295477032745c3b7b2839c77679705 [id=842fcbdfc3ea408ea71b929b54751160]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.034075 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46005e39ab284da6bf1bb9565318a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.037262 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 321efd02c5204013b253b43cacdd2e92 (DEFAULT_TABLE table=default.Table_c8707ebf10c24701a50991c2b06b50bd [id=3a6e4595bb934b6fbe8269e08d41ba68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.037649 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321efd02c5204013b253b43cacdd2e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.041646 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 (DEFAULT_TABLE table=default.Table_de40cf71b0974c799cf87de718e159b4 [id=a8b87d1e66ad42798ddfa4de05f5dbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.042018 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b92999890ee49d78d7c16f5ed1c4958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.043488 5514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap replayed 1/1 log segments. Stats: ops{read=116 overwritten=1 applied=113 ignored=0} inserts{seen=97 ignored=0} mutations{seen=115 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240503 02:12:26.047984 5514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Bootstrap complete.
I20240503 02:12:26.048641 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c7da59f96b8141f093a1b8de9a334e5d" } new_table_name: "default.Table_df82f492a3db4e1480369126b97c1169" modify_external_catalogs: true
I20240503 02:12:26.050037 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.050572 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.050788 5106 tablet_bootstrap.cc:654] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.051609 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3df92b6e544de9ca76d98706eebec (DEFAULT_TABLE table=default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8 [id=d636c15b30004d098fa523ef493335db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.051842 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 043ba14eaa5b46319b0b88457d2a5f45 deleted and replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5
I20240503 02:12:26.052039 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3df92b6e544de9ca76d98706eebec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.052239 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_828411f9b595429a8ca36e4b054c14f1" } new_table_name: "default.Table_da717b1e145649f3b0762f6ba74ae5f5" modify_external_catalogs: true
I20240503 02:12:26.053058 5114 ts_tablet_manager.cc:1888] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:26.053215 5114 log.cc:1196] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/959d998ec4ec4a52a6031650f2da4691
I20240503 02:12:26.055258 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
I20240503 02:12:26.055624 4269 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:26.058060 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 (DEFAULT_TABLE table=default.Table_a2295477032745c3b7b2839c77679705 [id=842fcbdfc3ea408ea71b929b54751160]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.059837 5115 ts_tablet_manager.cc:1888] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:26.061314 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 94559cfc32704eae96b121f165686edc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94559cfc32704eae96b121f165686edc already in progress: creating tablet
I20240503 02:12:26.061514 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 2a51563396614c54b980f0679e84a2b1 (table default.Table_5e9ba5798f4c4414826196e0761c2df9 [id=0a6dd433bee8497395826808bba7774e]) successfully deleted
I20240503 02:12:26.063302 4494 tablet_service.cc:1467] Processing CreateTablet for tablet b9470480a9de45209044f8e8c461519a (DEFAULT_TABLE table=default.Table_5916ed7fb62a4cdc910846e24178b911 [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.063957 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9470480a9de45209044f8e8c461519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.065598 5114 ts_tablet_manager.cc:1909] T 959d998ec4ec4a52a6031650f2da4691 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:26.066337 5114 tablet_replica.cc:330] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:26.060319 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 (DEFAULT_TABLE table=default.Table_de40cf71b0974c799cf87de718e159b4 [id=a8b87d1e66ad42798ddfa4de05f5dbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.062232 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 321efd02c5204013b253b43cacdd2e92 (DEFAULT_TABLE table=default.Table_c8707ebf10c24701a50991c2b06b50bd [id=3a6e4595bb934b6fbe8269e08d41ba68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.069584 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2ecfd29eddde4c39b7661b1b7d96a08e
I20240503 02:12:26.069799 5115 log.cc:1196] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/959d998ec4ec4a52a6031650f2da4691
I20240503 02:12:26.071393 4503 tablet_service.cc:1467] Processing CreateTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 (DEFAULT_TABLE table=default.Table_bfb8401dc518472abd4e26a2c788377d [id=10893601d3fa4c87b63e5c8acc68bce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.072111 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:26.072248 4636 tablet_service.cc:1467] Processing CreateTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 (DEFAULT_TABLE table=default.Table_bfb8401dc518472abd4e26a2c788377d [id=10893601d3fa4c87b63e5c8acc68bce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.072463 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:26.072824 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:26.062228 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 28c1a81589df496b92b980353f00e594 (DEFAULT_TABLE table=default.Table_6d070a69982c45b7904a12c6df0386b6 [id=6f362fd85655487d93025664e6a68fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.062546 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46005e39ab284da6bf1bb9565318a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.063412 4641 tablet_service.cc:1467] Processing CreateTablet for tablet b9470480a9de45209044f8e8c461519a (DEFAULT_TABLE table=default.Table_5916ed7fb62a4cdc910846e24178b911 [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.073918 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9470480a9de45209044f8e8c461519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.077261 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e (DEFAULT_TABLE table=default.Table_df799d52e6434e0f90b831960d0f39c1 [id=19ac7478bcac48f191daaf64dcecdec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.077383 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c1a81589df496b92b980353f00e594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.077716 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfd29eddde4c39b7661b1b7d96a08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.077850 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 857d019b8ab94be1b7719fec83e57e55 (DEFAULT_TABLE table=default.Table_c713f241c06a4a7c8106ba83bcab0c70 [id=ebbf977116684eaf914cb6c4807e73c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.078207 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857d019b8ab94be1b7719fec83e57e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.081279 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf (DEFAULT_TABLE table=default.Table_a567b7af96934be8836963c85cc81473 [id=9c42710fdba04850b53882a01803eeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.081629 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a082f4c43fd4c188be9ee3966e26edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.085191 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 50a43e428b804729adf9d8a3133c90ff (DEFAULT_TABLE table=default.Table_d689749a9c6a4c91a19f092325e08e73 [id=7aab2fb7ae69485da4ccef5840ae3aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.085628 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a43e428b804729adf9d8a3133c90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.085757 4647 tablet_service.cc:1467] Processing CreateTablet for tablet d6c63f889cb54619927ade6989fed63a (DEFAULT_TABLE table=default.Table_c5d56a6132f94935a81edadcf1f0d7f3 [id=c0a47dd6f6f54b9b8f752dc5a51ea8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.085183 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.086597 5114 raft_consensus.cc:2229] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:26.090029 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 905ca3be01e344d38350a58ac9f0d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.092105 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 50a43e428b804729adf9d8a3133c90ff (DEFAULT_TABLE table=default.Table_d689749a9c6a4c91a19f092325e08e73 [id=7aab2fb7ae69485da4ccef5840ae3aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.092703 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.093103 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.093474 4492 tablet_service.cc:1467] Processing CreateTablet for tablet d6c63f889cb54619927ade6989fed63a (DEFAULT_TABLE table=default.Table_c5d56a6132f94935a81edadcf1f0d7f3 [id=c0a47dd6f6f54b9b8f752dc5a51ea8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.093988 4498 tablet_service.cc:1467] Processing CreateTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb (DEFAULT_TABLE table=default.Table_93262e3fdea24977a7a3aa557373b9eb [id=501dab2edaf84e3aaceab4bd5984617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.094660 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 7a047360ed8843008c4ba633a65b2769 (DEFAULT_TABLE table=default.Table_5b3b2e6e211c47338aa5ddca26669a20 [id=f2bff3fc937040f7989f4c1e43c5773a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.095238 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 054d57b190ef42ba81f1c8a2cf00761b (DEFAULT_TABLE table=default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.095724 4493 tablet_service.cc:1467] Processing CreateTablet for tablet bfcc73e2eb57429aa61b52b42198c747 (DEFAULT_TABLE table=default.Table_15cc1a53bc594d6b8a04c4588fa6ec14 [id=7230242df5244c83974d492b97d838a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.096236 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 89b57a7235b546769c30745dd4c3b9f5 (DEFAULT_TABLE table=default.Table_029aae84b86047689e44978da354aefa [id=8ed2079ee5dd4f35ac77bd52af620d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.096700 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 04d555c8d8844ee49c960a5279485480 (DEFAULT_TABLE table=default.Table_f471b5f3094048268179ff4ff91b119f [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.097186 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 857d019b8ab94be1b7719fec83e57e55 (DEFAULT_TABLE table=default.Table_c713f241c06a4a7c8106ba83bcab0c70 [id=ebbf977116684eaf914cb6c4807e73c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.100765 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e (DEFAULT_TABLE table=default.Table_df799d52e6434e0f90b831960d0f39c1 [id=19ac7478bcac48f191daaf64dcecdec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.101141 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca2c34e0fc44be8b7ee9252ef557497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.101481 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d57b190ef42ba81f1c8a2cf00761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.102025 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfd29eddde4c39b7661b1b7d96a08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.103298 5103 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.103466 5103 ts_tablet_manager.cc:1367] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20240503 02:12:26.104336 5103 raft_consensus.cc:348] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.104547 5103 raft_consensus.cc:374] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.104629 5103 raft_consensus.cc:729] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.104940 5103 consensus_queue.cc:260] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.105224 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf (DEFAULT_TABLE table=default.Table_a567b7af96934be8836963c85cc81473 [id=9c42710fdba04850b53882a01803eeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.105571 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a082f4c43fd4c188be9ee3966e26edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.105818 5103 ts_tablet_manager.cc:1398] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:12:26.105979 5589 raft_consensus.cc:482] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.091874 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ea8c1983dad340faaa278bfe0d6ea103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.106456 5589 raft_consensus.cc:504] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.092378 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3f9994ca5aea48d3bc25c097dddba3c3
W20240503 02:12:26.098400 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b23be1160c454c139128aff5a960d183 consensus state has no leader
W20240503 02:12:26.099494 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf10c48d165047668fbd5ca17f917fe3 consensus state has no leader
I20240503 02:12:26.111982 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c3b4aa722ff64856923050d70b17d823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.063254 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd (DEFAULT_TABLE table=default.Table_ab30b57f7d9044fca1bf76dc5e2bc323 [id=b7bcb6c138184fdab9ffe2e84b76772c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.116158 5589 leader_election.cc:288] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:26.116426 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.116468 5103 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.121037 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.121714 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:26.125044 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 3db446d8a6984d9cb4175887f7e125bb (DEFAULT_TABLE table=default.Table_4101d1a1a4c14c018fecdd3b3d05641f [id=7a8979a4524b4879880b449477f750ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.130630 5114 raft_consensus.cc:2258] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.131119 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 905ca3be01e344d38350a58ac9f0d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.132944 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 50e04292742646049f39b6aa29338d75 (DEFAULT_TABLE table=default.Table_0134301a980f41e5847132881085032e [id=d0177c0786914f7cb381e9693c930937]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.133359 5114 ts_tablet_manager.cc:1875] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:26.154203 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4bb66ff27eed4eb9b2513d844a9bdbbf" } new_table_name: "default.Table_301d83f516e84342a226f385300cef79" modify_external_catalogs: true
I20240503 02:12:26.158368 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.158458 4638 tablet_service.cc:1467] Processing CreateTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb (DEFAULT_TABLE table=default.Table_93262e3fdea24977a7a3aa557373b9eb [id=501dab2edaf84e3aaceab4bd5984617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.158406 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.160605 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c63f889cb54619927ade6989fed63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.161612 5115 ts_tablet_manager.cc:1909] T 959d998ec4ec4a52a6031650f2da4691 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:26.162103 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 7a047360ed8843008c4ba633a65b2769 (DEFAULT_TABLE table=default.Table_5b3b2e6e211c47338aa5ddca26669a20 [id=f2bff3fc937040f7989f4c1e43c5773a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.164057 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 50e04292742646049f39b6aa29338d75 (DEFAULT_TABLE table=default.Table_0134301a980f41e5847132881085032e [id=d0177c0786914f7cb381e9693c930937]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.165307 5514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:26.167240 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d689749a9c6a4c91a19f092325e08e73" } new_table_name: "default.Table_01a2313a2c52424386a931aeecfac97c" modify_external_catalogs: true
W20240503 02:12:26.179338 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c2a8190210439fb7589cb02ec82bc2 consensus state has no leader
I20240503 02:12:26.183863 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_63c3234b89e94ea4a79400239439762d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.186493 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6bb9a205ce214c72a18394200f496a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.193662 5159 log.cc:1196] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2a51563396614c54b980f0679e84a2b1
I20240503 02:12:26.194803 5331 raft_consensus.cc:482] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.195024 5331 raft_consensus.cc:504] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.197263 5331 leader_election.cc:288] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:26.197980 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb9a205ce214c72a18394200f496a52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:26.198490 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c3234b89e94ea4a79400239439762d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:26.197962 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8c1983dad340faaa278bfe0d6ea103 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:26.200553 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3df92b6e544de9ca76d98706eebec (DEFAULT_TABLE table=default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8 [id=d636c15b30004d098fa523ef493335db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.200824 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.201777 5589 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.201959 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:26.202004 5589 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.202666 5589 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:26.201418 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3b4aa722ff64856923050d70b17d823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:26.202984 4811 raft_consensus.cc:2454] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:26.203568 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.209133 5159 ts_tablet_manager.cc:1909] T 2a51563396614c54b980f0679e84a2b1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:26.212160 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_997fff88119a49a5b51f25324fe4f546"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:26.213191 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 (DEFAULT_TABLE table=default.Table_a2295477032745c3b7b2839c77679705 [id=842fcbdfc3ea408ea71b929b54751160]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.214821 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 28c1a81589df496b92b980353f00e594 (DEFAULT_TABLE table=default.Table_6d070a69982c45b7904a12c6df0386b6 [id=6f362fd85655487d93025664e6a68fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.215276 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 76aef6b021da4f5aad7f523b15d484ec (DEFAULT_TABLE table=default.Table_4f5be79fd35347a7ac91d7cc508b4a67 [id=b078c65aa39343aabe02f29bf833abe6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:26.215601 4746 leader_election.cc:341] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:26.216352 4462 leader_election.cc:302] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:26.218478 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 054d57b190ef42ba81f1c8a2cf00761b (DEFAULT_TABLE table=default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.218734 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:26.221101 5589 raft_consensus.cc:2781] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.221226 5589 raft_consensus.cc:482] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.221318 5589 raft_consensus.cc:3037] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.063602 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 76aef6b021da4f5aad7f523b15d484ec (DEFAULT_TABLE table=default.Table_4f5be79fd35347a7ac91d7cc508b4a67 [id=b078c65aa39343aabe02f29bf833abe6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:26.212877 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_997fff88119a49a5b51f25324fe4f546 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:26.229316 5589 raft_consensus.cc:504] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.210693 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 321efd02c5204013b253b43cacdd2e92 (DEFAULT_TABLE table=default.Table_c8707ebf10c24701a50991c2b06b50bd [id=3a6e4595bb934b6fbe8269e08d41ba68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.232668 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3df92b6e544de9ca76d98706eebec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.234540 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d555c8d8844ee49c960a5279485480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.235078 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db446d8a6984d9cb4175887f7e125bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.238250 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 382) took 1313 ms. Trace:
I20240503 02:12:26.238369 5159 rpcz_store.cc:276] 0503 02:12:24.924936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.924987 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:25.888972 (+963985us) ts_tablet_manager.cc:1120] Deleting tablet 2a51563396614c54b980f0679e84a2b1
0503 02:12:25.888981 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:25.970543 (+ 81562us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:26.008832 (+ 38289us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:26.238180 (+229348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963745}
I20240503 02:12:26.238845 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 383) took 1313 ms. Trace:
I20240503 02:12:26.238940 5159 rpcz_store.cc:276] 0503 02:12:24.925484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.925533 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:26.238676 (+1313143us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:26.238685 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.238795 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1312957}
I20240503 02:12:26.239598 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46005e39ab284da6bf1bb9565318a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.241184 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a047360ed8843008c4ba633a65b2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.245327 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b57a7235b546769c30745dd4c3b9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.245524 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c1a81589df496b92b980353f00e594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.246042 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa77e6c9a494fa2bdc74c5d725624cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.250309 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aef6b021da4f5aad7f523b15d484ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.253883 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321efd02c5204013b253b43cacdd2e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.256381 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c63f889cb54619927ade6989fed63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.261533 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e04292742646049f39b6aa29338d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.234966 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd (DEFAULT_TABLE table=default.Table_ab30b57f7d9044fca1bf76dc5e2bc323 [id=b7bcb6c138184fdab9ffe2e84b76772c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.270673 4784 tablet_service.cc:1467] Processing CreateTablet for tablet b9470480a9de45209044f8e8c461519a (DEFAULT_TABLE table=default.Table_5916ed7fb62a4cdc910846e24178b911 [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.272382 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e (DEFAULT_TABLE table=default.Table_df799d52e6434e0f90b831960d0f39c1 [id=19ac7478bcac48f191daaf64dcecdec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.272926 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf (DEFAULT_TABLE table=default.Table_a567b7af96934be8836963c85cc81473 [id=9c42710fdba04850b53882a01803eeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.273419 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 50a43e428b804729adf9d8a3133c90ff (DEFAULT_TABLE table=default.Table_d689749a9c6a4c91a19f092325e08e73 [id=7aab2fb7ae69485da4ccef5840ae3aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.276580 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.276643 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:26.281562 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]). Expected version 1 got 0
I20240503 02:12:26.282100 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]). Expected version 3 got 0
I20240503 02:12:26.282442 4274 catalog_manager.cc:5484] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:26.282987 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:26.283262 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:26.283516 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 50e04292742646049f39b6aa29338d75 (DEFAULT_TABLE table=default.Table_0134301a980f41e5847132881085032e [id=d0177c0786914f7cb381e9693c930937]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.283587 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.284428 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a43e428b804729adf9d8a3133c90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.285861 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 905ca3be01e344d38350a58ac9f0d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.286712 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9470480a9de45209044f8e8c461519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.287667 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e04292742646049f39b6aa29338d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.288779 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.288833 5105 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:26.289211 5106 tablet_bootstrap.cc:492] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:26.289530 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a082f4c43fd4c188be9ee3966e26edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.290598 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfd29eddde4c39b7661b1b7d96a08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.291626 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8241a88f5c3a47578c92343f9fb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.292392 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.292678 4671 raft_consensus.cc:2454] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:26.293506 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:26.294185 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:26.294235 4644 tablet_service.cc:1467] Processing CreateTablet for tablet bfcc73e2eb57429aa61b52b42198c747 (DEFAULT_TABLE table=default.Table_15cc1a53bc594d6b8a04c4588fa6ec14 [id=7230242df5244c83974d492b97d838a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.294741 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857d019b8ab94be1b7719fec83e57e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.294682 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 3db446d8a6984d9cb4175887f7e125bb (DEFAULT_TABLE table=default.Table_4101d1a1a4c14c018fecdd3b3d05641f [id=7a8979a4524b4879880b449477f750ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.295012 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 89b57a7235b546769c30745dd4c3b9f5 (DEFAULT_TABLE table=default.Table_029aae84b86047689e44978da354aefa [id=8ed2079ee5dd4f35ac77bd52af620d00]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:26.296554 4462 leader_election.cc:341] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:26.298669 4608 consensus_peers.cc:485] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:26.299060 5479 consensus_queue.cc:1035] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:12:26.300961 5106 ts_tablet_manager.cc:1367] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.287s user 0.012s sys 0.027s
I20240503 02:12:26.303078 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 04d555c8d8844ee49c960a5279485480 (DEFAULT_TABLE table=default.Table_f471b5f3094048268179ff4ff91b119f [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.309250 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcc73e2eb57429aa61b52b42198c747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.295418 4811 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
W20240503 02:12:26.298494 4746 leader_election.cc:341] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:26.299835 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 7a047360ed8843008c4ba633a65b2769 (DEFAULT_TABLE table=default.Table_5b3b2e6e211c47338aa5ddca26669a20 [id=f2bff3fc937040f7989f4c1e43c5773a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.300410 4788 tablet_service.cc:1467] Processing CreateTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb (DEFAULT_TABLE table=default.Table_93262e3fdea24977a7a3aa557373b9eb [id=501dab2edaf84e3aaceab4bd5984617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.298799 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 384) took 1342 ms. Trace:
I20240503 02:12:26.300684 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 3db446d8a6984d9cb4175887f7e125bb (DEFAULT_TABLE table=default.Table_4101d1a1a4c14c018fecdd3b3d05641f [id=7a8979a4524b4879880b449477f750ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.301393 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 054d57b190ef42ba81f1c8a2cf00761b (DEFAULT_TABLE table=default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.302433 4776 tablet_service.cc:1467] Processing CreateTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 (DEFAULT_TABLE table=default.Table_bfb8401dc518472abd4e26a2c788377d [id=10893601d3fa4c87b63e5c8acc68bce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.300716 4791 tablet_service.cc:1467] Processing CreateTablet for tablet bfcc73e2eb57429aa61b52b42198c747 (DEFAULT_TABLE table=default.Table_15cc1a53bc594d6b8a04c4588fa6ec14 [id=7230242df5244c83974d492b97d838a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.316340 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.316586 5115 tablet_replica.cc:330] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:26.316833 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 04d555c8d8844ee49c960a5279485480 (DEFAULT_TABLE table=default.Table_f471b5f3094048268179ff4ff91b119f [id=f95b06fca203484c90903ffc1a2dd58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.317065 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 89b57a7235b546769c30745dd4c3b9f5 (DEFAULT_TABLE table=default.Table_029aae84b86047689e44978da354aefa [id=8ed2079ee5dd4f35ac77bd52af620d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.317253 4746 leader_election.cc:302] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:26.320199 5331 raft_consensus.cc:2726] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:26.320158 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 (DEFAULT_TABLE table=default.Table_55379d2e66cd42d28abf6943d13ca28e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.321009 4462 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:26.321204 5106 raft_consensus.cc:348] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.321635 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8241a88f5c3a47578c92343f9fb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.322211 5286 raft_consensus.cc:2781] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.322376 5106 raft_consensus.cc:374] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.322522 5106 raft_consensus.cc:729] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.322751 5286 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.322891 5286 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.325838 5106 consensus_queue.cc:260] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.326527 5106 ts_tablet_manager.cc:1398] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 02:12:26.338866 5106 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:26.348134 5286 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.349023 5286 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:26.349925 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:26.354096 5159 rpcz_store.cc:276] 0503 02:12:24.933837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.933886 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:26.239299 (+1305413us) ts_tablet_manager.cc:1120] Deleting tablet 2152f205cf1b4ec7ae03605bbfa007e1
0503 02:12:26.239307 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.275862 (+ 36555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305175}
W20240503 02:12:26.354741 4462 leader_election.cc:341] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:26.354777 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 386) took 1302 ms. Trace:
I20240503 02:12:26.358067 5103 tablet_bootstrap.cc:654] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.316253 5514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c621d795c44d0daa534261920d874a, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.364756 5114 ts_tablet_manager.cc:1888] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:26.364950 5114 log.cc:1196] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/46560ac6d94943d2a8fb3b3904f2b5b6
I20240503 02:12:26.354876 5159 rpcz_store.cc:276] 0503 02:12:25.051837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.051905 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:26.354580 (+1302675us) ts_tablet_manager.cc:1120] Deleting tablet ff8d2c0991e849b5afa35e7230db298d
0503 02:12:26.354595 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.354721 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302446}
I20240503 02:12:26.369653 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 385) took 1318 ms. Trace:
W20240503 02:12:26.369125 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 48902.7us
I20240503 02:12:26.370327 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:26.370596 4811 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.372745 5114 ts_tablet_manager.cc:1909] T 46560ac6d94943d2a8fb3b3904f2b5b6 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:26.373253 5159 rpcz_store.cc:276] 0503 02:12:25.050772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.052170 (+ 1398us) service_pool.cc:225] Handling call
0503 02:12:26.369431 (+1317261us) ts_tablet_manager.cc:1120] Deleting tablet aa699a1dc3c34be59925560d5e11f49e
0503 02:12:26.369449 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.369598 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317075}
I20240503 02:12:26.373775 5159 tablet_replica.cc:330] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:26.374053 5159 raft_consensus.cc:2229] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:26.375080 5114 tablet_replica.cc:330] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:26.375425 5114 raft_consensus.cc:2229] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:26.375870 5114 pending_rounds.cc:62] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:26.375998 5114 pending_rounds.cc:68] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420807943847936 op_type: NO_OP noop_request { }
I20240503 02:12:26.376173 5114 raft_consensus.cc:2866] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:26.376329 5114 raft_consensus.cc:2258] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.377490 5114 ts_tablet_manager.cc:1875] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:26.377229 5159 raft_consensus.cc:2258] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.377532 5514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 113, Last appended: 5.115, Last appended by leader: 115, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:26.382728 4811 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:26.384291 4462 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:26.385226 5286 raft_consensus.cc:2781] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.385385 5286 raft_consensus.cc:686] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:26.385710 5286 consensus_queue.cc:237] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.422989 5103 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.423182 5103 ts_tablet_manager.cc:1367] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.317s user 0.013s sys 0.000s
I20240503 02:12:26.424083 5103 raft_consensus.cc:348] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.424757 5103 raft_consensus.cc:374] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.424909 5103 raft_consensus.cc:729] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.425343 5103 consensus_queue.cc:260] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.426105 5103 ts_tablet_manager.cc:1398] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:26.432679 5103 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.434792 5103 tablet_bootstrap.cc:654] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.435868 5577 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.436090 5577 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.436852 5577 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:26.437762 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:26.438047 4528 raft_consensus.cc:2454] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:26.438549 4608 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:26.438998 5577 raft_consensus.cc:2781] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.294096 5105 ts_tablet_manager.cc:1367] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.330s user 0.033s sys 0.031s
I20240503 02:12:26.439184 5577 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.439322 5577 raft_consensus.cc:3037] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.442760 5577 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.443675 5106 tablet_bootstrap.cc:654] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.444891 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.447749 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa77e6c9a494fa2bdc74c5d725624cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.452826 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d555c8d8844ee49c960a5279485480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.450963 5577 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:26.453507 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d57b190ef42ba81f1c8a2cf00761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.459483 5589 leader_election.cc:288] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:26.460054 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:26.460542 4670 raft_consensus.cc:3037] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.460944 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db446d8a6984d9cb4175887f7e125bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.463640 5426 raft_consensus.cc:482] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.464367 4670 raft_consensus.cc:2454] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:26.474720 5159 ts_tablet_manager.cc:1875] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:26.476384 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 857d019b8ab94be1b7719fec83e57e55 (DEFAULT_TABLE table=default.Table_c713f241c06a4a7c8106ba83bcab0c70 [id=ebbf977116684eaf914cb6c4807e73c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.478518 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a43e428b804729adf9d8a3133c90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.482101 4787 tablet_service.cc:1467] Processing CreateTablet for tablet d6c63f889cb54619927ade6989fed63a (DEFAULT_TABLE table=default.Table_c5d56a6132f94935a81edadcf1f0d7f3 [id=c0a47dd6f6f54b9b8f752dc5a51ea8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.482643 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857d019b8ab94be1b7719fec83e57e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.483620 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c63f889cb54619927ade6989fed63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.487115 5106 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:26.487280 5106 ts_tablet_manager.cc:1367] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.149s user 0.001s sys 0.014s
I20240503 02:12:26.487488 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 (DEFAULT_TABLE table=default.Table_de40cf71b0974c799cf87de718e159b4 [id=a8b87d1e66ad42798ddfa4de05f5dbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.488112 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b57a7235b546769c30745dd4c3b9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.467737 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:26.468533 4462 leader_election.cc:302] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:26.492004 5426 raft_consensus.cc:504] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.492868 5426 leader_election.cc:288] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:26.495443 5114 ts_tablet_manager.cc:1888] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:26.495745 5114 log.cc:1196] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
I20240503 02:12:26.496366 5114 ts_tablet_manager.cc:1909] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:26.488685 5106 raft_consensus.cc:348] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.497201 5106 raft_consensus.cc:374] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.497792 5106 raft_consensus.cc:729] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.497882 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:26.498113 5106 consensus_queue.cc:260] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.499001 5106 ts_tablet_manager.cc:1398] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:12:26.499114 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321efd02c5204013b253b43cacdd2e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.499356 5106 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:26.501791 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 42e80cc31f5c4908b82f2b030255f6af (table default.Table_be851867e140416c8921098eaa348201 [id=04d5d63e679c47ce9a72f1ffeeeaf506], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 42e80cc31f5c4908b82f2b030255f6af consensus state has no leader
I20240503 02:12:26.502611 5106 tablet_bootstrap.cc:654] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.503701 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e04292742646049f39b6aa29338d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.503959 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcc73e2eb57429aa61b52b42198c747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.505172 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:26.505759 4529 raft_consensus.cc:3037] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.507570 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa77e6c9a494fa2bdc74c5d725624cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:26.508972 4746 leader_election.cc:341] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:26.513384 4528 raft_consensus.cc:2454] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:26.514060 5589 raft_consensus.cc:2781] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.514542 5589 raft_consensus.cc:686] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:26.514963 4746 leader_election.cc:302] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:26.515100 5589 consensus_queue.cc:237] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.516721 5426 raft_consensus.cc:2781] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.516884 5426 raft_consensus.cc:482] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.517314 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdefb33fe6e4a9893dd66f6252395d7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:26.517719 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a047360ed8843008c4ba633a65b2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.518419 5426 raft_consensus.cc:3037] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.520485 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcc73e2eb57429aa61b52b42198c747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.523555 4811 raft_consensus.cc:3037] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.524464 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b57a7235b546769c30745dd4c3b9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.539399 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d57b190ef42ba81f1c8a2cf00761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.540357 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d555c8d8844ee49c960a5279485480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.542979 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:26.076908 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 959d998ec4ec4a52a6031650f2da4691 (table default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]) successfully deleted
I20240503 02:12:26.553781 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd (DEFAULT_TABLE table=default.Table_ab30b57f7d9044fca1bf76dc5e2bc323 [id=b7bcb6c138184fdab9ffe2e84b76772c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.555305 5426 raft_consensus.cc:504] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.556090 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db446d8a6984d9cb4175887f7e125bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.525933 5105 raft_consensus.cc:348] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.557508 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:26.519980 5514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:26.560853 4812 raft_consensus.cc:2454] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:26.561547 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:26.561893 4528 raft_consensus.cc:3037] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.562235 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca2c34e0fc44be8b7ee9252ef557497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.564594 5105 raft_consensus.cc:374] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.566267 4528 raft_consensus.cc:2454] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:26.566654 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b92999890ee49d78d7c16f5ed1c4958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.568174 4746 leader_election.cc:302] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:26.570011 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 493070us
I20240503 02:12:26.570961 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.571645 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 (DEFAULT_TABLE table=default.Table_55379d2e66cd42d28abf6943d13ca28e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.572233 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.572427 4506 tablet_service.cc:1467] Processing CreateTablet for tablet cbd9c89388354e87a56a2c38d791bff8 (DEFAULT_TABLE table=default.Table_2771f168295e4bcb84010eb642198fa8 [id=67f723bc7fae405684113405f6162e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.572816 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd9c89388354e87a56a2c38d791bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.573305 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 620204f7cb35442f87eae45f84beeee1 (DEFAULT_TABLE table=default.Table_b8df46519f1a401b862bbfabff3577fe [id=27c7348bc7034ab8b9a4f483134459a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.573901 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620204f7cb35442f87eae45f84beeee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.580986 5331 raft_consensus.cc:2781] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.582864 5331 raft_consensus.cc:686] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:26.583832 5426 leader_election.cc:288] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:26.585832 5105 raft_consensus.cc:729] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.588124 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8241a88f5c3a47578c92343f9fb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.589008 5331 consensus_queue.cc:237] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.601313 5103 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.603744 5103 ts_tablet_manager.cc:1367] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.171s user 0.012s sys 0.000s
I20240503 02:12:26.607381 4811 raft_consensus.cc:2454] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:26.594460 5105 consensus_queue.cc:260] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.572053 4529 raft_consensus.cc:2454] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:26.618744 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.619601 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 620204f7cb35442f87eae45f84beeee1 (DEFAULT_TABLE table=default.Table_b8df46519f1a401b862bbfabff3577fe [id=27c7348bc7034ab8b9a4f483134459a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.622691 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 2a51563396614c54b980f0679e84a2b1 (table default.Table_5e9ba5798f4c4414826196e0761c2df9 [id=0a6dd433bee8497395826808bba7774e]) successfully deleted
W20240503 02:12:26.623106 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6a3ca705e8482a8cd6a6fb17d93470 already in progress: creating tablet
W20240503 02:12:26.623895 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ff8d2c0991e849b5afa35e7230db298d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff8d2c0991e849b5afa35e7230db298d already in progress: creating tablet
W20240503 02:12:26.624578 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet aa699a1dc3c34be59925560d5e11f49e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa699a1dc3c34be59925560d5e11f49e already in progress: creating tablet
I20240503 02:12:26.630851 4506 tablet_service.cc:1467] Processing CreateTablet for tablet dc16522e513c44129f4bbad7c31edb9a (DEFAULT_TABLE table=default.Table_0a16926a4afd4923bed405fe42c0a348 [id=f448d6b54faa4aa7b48ef70c3fa009b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.630913 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 368aabce19984d03aa7116c8df990ef1 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.631057 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:26.635299 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 (DEFAULT_TABLE table=default.Table_55379d2e66cd42d28abf6943d13ca28e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.641148 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620204f7cb35442f87eae45f84beeee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.651852 4787 tablet_service.cc:1467] Processing CreateTablet for tablet cbd9c89388354e87a56a2c38d791bff8 (DEFAULT_TABLE table=default.Table_2771f168295e4bcb84010eb642198fa8 [id=67f723bc7fae405684113405f6162e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.652892 5105 ts_tablet_manager.cc:1398] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.182s user 0.003s sys 0.001s
I20240503 02:12:26.657583 4780 tablet_service.cc:1467] Processing CreateTablet for tablet dc16522e513c44129f4bbad7c31edb9a (DEFAULT_TABLE table=default.Table_0a16926a4afd4923bed405fe42c0a348 [id=f448d6b54faa4aa7b48ef70c3fa009b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.661032 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:26.663954 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.664602 5105 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:26.671363 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 368aabce19984d03aa7116c8df990ef1 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.692381 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368aabce19984d03aa7116c8df990ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.696698 4575 heartbeater.cc:347] Connected to a master server at 127.4.17.126:43321
I20240503 02:12:26.702457 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc16522e513c44129f4bbad7c31edb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.738405 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc16522e513c44129f4bbad7c31edb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.745608 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd9c89388354e87a56a2c38d791bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.749953 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368aabce19984d03aa7116c8df990ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.760527 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:26.760568 5382 consensus_queue.cc:1035] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240503 02:12:26.764164 4608 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:26.764562 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b92999890ee49d78d7c16f5ed1c4958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.769044 5103 raft_consensus.cc:348] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.776961 5103 raft_consensus.cc:374] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.778738 5103 raft_consensus.cc:729] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.776022 5382 raft_consensus.cc:2781] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.781050 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca2c34e0fc44be8b7ee9252ef557497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.782012 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:26.782155 5579 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.004s sys 0.010s
I20240503 02:12:26.783654 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 620204f7cb35442f87eae45f84beeee1 (DEFAULT_TABLE table=default.Table_b8df46519f1a401b862bbfabff3577fe [id=27c7348bc7034ab8b9a4f483134459a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.787146 5598 consensus_queue.cc:1035] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:26.786907 5103 consensus_queue.cc:260] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.789525 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 959d998ec4ec4a52a6031650f2da4691 (table default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]) successfully deleted
W20240503 02:12:26.789989 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 79b8db679ba24ac6a93be591c0513ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79b8db679ba24ac6a93be591c0513ab0 already in progress: creating tablet
I20240503 02:12:26.790407 5382 raft_consensus.cc:686] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:26.790774 5382 consensus_queue.cc:237] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.791574 5106 tablet_bootstrap.cc:492] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:26.793231 5106 ts_tablet_manager.cc:1367] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.292s user 0.014s sys 0.023s
W20240503 02:12:26.793893 5579 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.004s sys 0.011s
I20240503 02:12:26.795935 4652 tablet_service.cc:1467] Processing CreateTablet for tablet cbd9c89388354e87a56a2c38d791bff8 (DEFAULT_TABLE table=default.Table_2771f168295e4bcb84010eb642198fa8 [id=67f723bc7fae405684113405f6162e5b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:26.796907 4746 leader_election.cc:341] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:26.798839 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 368aabce19984d03aa7116c8df990ef1 (DEFAULT_TABLE table=default.Table_174379bd4f48483fa1baddfe455090de [id=aef6e606e6f34ed78039dd362329ce77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.799438 5115 raft_consensus.cc:2229] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:26.799501 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:26.804265 5115 raft_consensus.cc:2258] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.806087 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aef6b021da4f5aad7f523b15d484ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.805615 5103 ts_tablet_manager.cc:1398] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.199s user 0.012s sys 0.000s
I20240503 02:12:26.810075 5115 ts_tablet_manager.cc:1875] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:26.813503 4641 tablet_service.cc:1467] Processing CreateTablet for tablet dc16522e513c44129f4bbad7c31edb9a (DEFAULT_TABLE table=default.Table_0a16926a4afd4923bed405fe42c0a348 [id=f448d6b54faa4aa7b48ef70c3fa009b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.814605 5106 raft_consensus.cc:348] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.814934 5106 raft_consensus.cc:374] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.815053 5106 raft_consensus.cc:729] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.815436 5106 consensus_queue.cc:260] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.817099 5106 ts_tablet_manager.cc:1398] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.024s user 0.002s sys 0.002s
I20240503 02:12:26.820011 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd9c89388354e87a56a2c38d791bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.826026 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc16522e513c44129f4bbad7c31edb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.827260 5105 tablet_bootstrap.cc:654] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.827431 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368aabce19984d03aa7116c8df990ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.829488 5589 consensus_queue.cc:1035] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:26.835842 5103 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.838454 5103 tablet_bootstrap.cc:654] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.845286 5103 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.845512 5103 ts_tablet_manager.cc:1367] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:12:26.847577 5103 raft_consensus.cc:348] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.847841 5103 raft_consensus.cc:374] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.847923 5103 raft_consensus.cc:729] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.848244 5103 consensus_queue.cc:260] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.848412 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a047360ed8843008c4ba633a65b2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.848794 5103 ts_tablet_manager.cc:1398] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:26.849195 5103 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.849437 5577 raft_consensus.cc:482] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.851001 5103 tablet_bootstrap.cc:654] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.852000 5106 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:26.856555 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620204f7cb35442f87eae45f84beeee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.858042 5103 tablet_bootstrap.cc:492] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.858235 5103 ts_tablet_manager.cc:1367] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:26.856310 5577 raft_consensus.cc:504] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.861485 5103 raft_consensus.cc:348] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.861807 5103 raft_consensus.cc:374] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.861934 5103 raft_consensus.cc:729] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.862234 5103 consensus_queue.cc:260] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.862871 5103 ts_tablet_manager.cc:1398] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:26.865378 5103 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.865389 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:26.865820 4529 raft_consensus.cc:2379] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:26.867615 5577 leader_election.cc:288] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:26.871381 5103 tablet_bootstrap.cc:654] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.878753 5103 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.878962 5103 ts_tablet_manager.cc:1367] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 02:12:26.879866 5103 raft_consensus.cc:348] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.880223 5103 raft_consensus.cc:374] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.880302 5103 raft_consensus.cc:729] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.880576 5103 consensus_queue.cc:260] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.881179 5103 ts_tablet_manager.cc:1398] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:26.881500 5103 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.883318 5103 tablet_bootstrap.cc:654] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.886163 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05d88a5c49b43bea6d46feccaaa6920" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:26.888054 5286 consensus_queue.cc:1035] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:26.889675 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 99725.3us
W20240503 02:12:26.890027 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ea5e4db5204e4acf8719100d02d533fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5e4db5204e4acf8719100d02d533fb already in progress: creating tablet
I20240503 02:12:26.879144 4269 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7] alter complete (version 2)
I20240503 02:12:26.892174 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]) successfully deleted
I20240503 02:12:26.892267 5103 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.892432 5103 ts_tablet_manager.cc:1367] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
W20240503 02:12:26.892696 4269 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 86) took 865 ms (client timeout 1000 ms). Trace:
W20240503 02:12:26.892796 4269 rpcz_store.cc:269] 0503 02:12:26.026876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.027465 (+ 589us) service_pool.cc:225] Handling call
0503 02:12:26.892630 (+865165us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:26.190576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:26.654418 (+463842us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:26.654467 (+ 49us) write_op.cc:435] Acquired schema lock
0503 02:12:26.654472 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:26.656980 (+ 2508us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:26.657003 (+ 23us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:26.657012 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:26.662891 (+ 5879us) tablet.cc:699] Row locks acquired
0503 02:12:26.662895 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:26.665620 (+ 2725us) write_op.cc:270] Start()
0503 02:12:26.665672 (+ 52us) write_op.cc:276] Timestamp: P: 1714702346662915 usec, L: 0
0503 02:12:26.665677 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:26.665927 (+ 250us) log.cc:842] Serialized 356 byte log entry
0503 02:12:26.792977 (+127050us) op_driver.cc:444] REPLICATION: finished
0503 02:12:26.878363 (+ 85386us) write_op.cc:301] APPLY: starting
0503 02:12:26.878383 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:26.878421 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:26.878464 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:26.878523 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:26.878562 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:26.878570 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:26.878665 (+ 95us) log.cc:842] Serialized 23 byte log entry
0503 02:12:26.878728 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:26.878745 (+ 17us) write_op.cc:454] Released schema lock
0503 02:12:26.878751 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:26.878817 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:26.878820 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1712,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":85297,"apply.run_cpu_time_us":527,"apply.run_wall_time_us":527,"num_ops":1,"prepare.queue_time_us":131897,"prepare.run_cpu_time_us":2021,"prepare.run_wall_time_us":533115,"raft.queue_time_us":4653,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":386,"replication_time_us":127232,"thread_start_us":86234,"threads_started":1}]]}
W20240503 02:12:26.880908 4268 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.66:56851 (request call id 66) took 818 ms (client timeout 1000 ms). Trace:
W20240503 02:12:26.894754 4268 rpcz_store.cc:269] 0503 02:12:26.062034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.070102 (+ 8068us) service_pool.cc:225] Handling call
0503 02:12:26.880858 (+810756us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20240503 02:12:26.895241 5103 raft_consensus.cc:348] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.895504 5103 raft_consensus.cc:374] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.895597 5103 raft_consensus.cc:729] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.895840 5103 consensus_queue.cc:260] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.896339 5103 ts_tablet_manager.cc:1398] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:26.900187 4608 leader_election.cc:302] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:26.905787 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_252b426637d647128fd41a25217a1cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:26.906649 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_252b426637d647128fd41a25217a1cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:26.908636 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8eb9b75d260a42229a49d6343e0b2e3a" } modify_external_catalogs: true
I20240503 02:12:26.908984 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8eb9b75d260a42229a49d6343e0b2e3a" } modify_external_catalogs: true
I20240503 02:12:26.911096 5479 raft_consensus.cc:3037] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.930764 5106 tablet_bootstrap.cc:654] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.934531 5103 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.936527 5103 tablet_bootstrap.cc:654] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:26.941457 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 49297.9us
W20240503 02:12:26.941954 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ea5e4db5204e4acf8719100d02d533fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5e4db5204e4acf8719100d02d533fb already in progress: creating tablet
W20240503 02:12:26.942337 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 79b8db679ba24ac6a93be591c0513ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79b8db679ba24ac6a93be591c0513ab0 already in progress: creating tablet
W20240503 02:12:26.943167 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 94559cfc32704eae96b121f165686edc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94559cfc32704eae96b121f165686edc already in progress: creating tablet
I20240503 02:12:26.955515 5103 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:26.955729 5103 ts_tablet_manager.cc:1367] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.005s
I20240503 02:12:26.956578 5103 raft_consensus.cc:348] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:26.956876 5103 raft_consensus.cc:374] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.956992 5103 raft_consensus.cc:729] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.957314 5103 consensus_queue.cc:260] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:26.957823 5626 log.cc:924] Time spent T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240503 02:12:26.957893 5103 ts_tablet_manager.cc:1398] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:26.958209 5103 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:26.960291 5103 tablet_bootstrap.cc:654] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.963867 5479 raft_consensus.cc:2726] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:26.968171 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e1c2fa9cb3481dafb9d7f1e826a30c already in progress: creating tablet
W20240503 02:12:26.968681 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3db94fb9d53044d796e2973143d888fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db94fb9d53044d796e2973143d888fb already in progress: creating tablet
I20240503 02:12:26.970520 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (table default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]) successfully deleted
W20240503 02:12:26.971616 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1436dd3df7284f9bb379236eb9d76836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1436dd3df7284f9bb379236eb9d76836 already in progress: creating tablet
W20240503 02:12:27.001708 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 043ba14eaa5b46319b0b88457d2a5f45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 043ba14eaa5b46319b0b88457d2a5f45 already in progress: creating tablet
W20240503 02:12:27.002171 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet cd66f88a24b74f2c9b2050af081d4601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd66f88a24b74f2c9b2050af081d4601 already in progress: creating tablet
W20240503 02:12:27.002470 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3aad7d6dcf2e4009872ce156c536f38e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aad7d6dcf2e4009872ce156c536f38e already in progress: creating tablet
W20240503 02:12:27.002843 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 905ca3be01e344d38350a58ac9f0d321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905ca3be01e344d38350a58ac9f0d321 already in progress: creating tablet
W20240503 02:12:27.003203 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2ecfd29eddde4c39b7661b1b7d96a08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ecfd29eddde4c39b7661b1b7d96a08e already in progress: creating tablet
W20240503 02:12:27.003734 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3f9994ca5aea48d3bc25c097dddba3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f9994ca5aea48d3bc25c097dddba3c3 already in progress: creating tablet
I20240503 02:12:27.007386 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.020371 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.020342 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.021706 5115 ts_tablet_manager.cc:1888] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:27.021872 5115 log.cc:1196] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/46560ac6d94943d2a8fb3b3904f2b5b6
I20240503 02:12:27.022058 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 959d998ec4ec4a52a6031650f2da4691 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.024827 5106 tablet_bootstrap.cc:492] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:27.025848 5115 ts_tablet_manager.cc:1909] T 46560ac6d94943d2a8fb3b3904f2b5b6 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:27.026221 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 400) took 1136 ms. Trace:
I20240503 02:12:27.029006 5115 rpcz_store.cc:276] 0503 02:12:25.890320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.890359 (+ 39us) service_pool.cc:225] Handling call
0503 02:12:26.292482 (+402123us) ts_tablet_manager.cc:1120] Deleting tablet 46560ac6d94943d2a8fb3b3904f2b5b6
0503 02:12:26.292503 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.799713 (+507210us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x6160001a0854. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9974ba00 00007f1c9aa25b83 00007f1c9d4b957c 00007f1c9d4eb8d4 00007f1c9d4ccbc0 00007f1c9d4cc94d 0000000000a60bcc 00007f1c8df23b74 00007f1c8df2719b 00007f1c8df27018
0503 02:12:26.948984 (+149271us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.013061 (+ 64077us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.026176 (+ 13115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1097845632,"tablet-delete.queue_time_us":401251}
I20240503 02:12:27.030133 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 401) took 1001 ms. Trace:
I20240503 02:12:27.030555 5115 rpcz_store.cc:276] 0503 02:12:26.029150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.041112 (+ 11962us) service_pool.cc:225] Handling call
0503 02:12:27.029923 (+988811us) ts_tablet_manager.cc:1120] Deleting tablet a0e1c2fa9cb3481dafb9d7f1e826a30c
0503 02:12:27.029939 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.030082 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":979692}
I20240503 02:12:27.037322 5115 tablet_replica.cc:330] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:27.037765 5115 raft_consensus.cc:2229] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:27.038182 5115 raft_consensus.cc:2258] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.039224 5115 ts_tablet_manager.cc:1875] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.042084 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]) successfully deleted
W20240503 02:12:27.042466 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e1c2fa9cb3481dafb9d7f1e826a30c already in progress: creating tablet
W20240503 02:12:27.042707 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3db94fb9d53044d796e2973143d888fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db94fb9d53044d796e2973143d888fb already in progress: creating tablet
W20240503 02:12:27.042941 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet cd66f88a24b74f2c9b2050af081d4601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd66f88a24b74f2c9b2050af081d4601 already in progress: creating tablet
W20240503 02:12:27.043169 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 905ca3be01e344d38350a58ac9f0d321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905ca3be01e344d38350a58ac9f0d321 already in progress: creating tablet
I20240503 02:12:27.045717 5106 ts_tablet_manager.cc:1367] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.201s user 0.005s sys 0.006s
I20240503 02:12:27.055697 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:27.059410 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0af50eb2b1c949a0a6a71e64a894e8a5 has no consensus state
W20240503 02:12:27.060470 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50a43e428b804729adf9d8a3133c90ff consensus state has no leader
I20240503 02:12:27.065748 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2c388fbe8fd148dc93f0d292381c0363" } modify_external_catalogs: true
I20240503 02:12:27.066015 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2c388fbe8fd148dc93f0d292381c0363" } modify_external_catalogs: true
I20240503 02:12:27.069159 4275 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 598) took 1039 ms. Trace:
I20240503 02:12:27.069290 4275 rpcz_store.cc:276] 0503 02:12:26.029894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.091652 (+ 61758us) service_pool.cc:225] Handling call
0503 02:12:26.199348 (+107696us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:26.199752 (+ 404us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:27.068863 (+869111us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:27.069010 (+ 147us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:27.069087 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:26.904337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:26.904483 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:26.904490 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:26.904492 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:26.904604 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:26.904617 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:26.904622 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:26.904713 (+ 91us) tablet.cc:699] Row locks acquired
0503 02:12:26.904715 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:26.904754 (+ 39us) write_op.cc:270] Start()
0503 02:12:26.904789 (+ 35us) write_op.cc:276] Timestamp: P: 1714702346904706 usec, L: 0
0503 02:12:26.904792 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:26.905070 (+ 278us) log.cc:842] Serialized 433 byte log entry
0503 02:12:27.068327 (+163257us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.068397 (+ 70us) write_op.cc:301] APPLY: starting
0503 02:12:27.068409 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.068448 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.068479 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.068561 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.068588 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:27.068600 (+ 12us) write_op.cc:312] APPLY: finished
0503 02:12:27.068660 (+ 60us) log.cc:842] Serialized 29 byte log entry
0503 02:12:27.068717 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.068730 (+ 13us) write_op.cc:454] Released schema lock
0503 02:12:27.068736 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:27.068776 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.068779 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":415,"num_ops":2,"prepare.queue_time_us":704188,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":820,"replication_time_us":163518,"spinlock_wait_cycles":152960}]]}
W20240503 02:12:27.071868 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1436dd3df7284f9bb379236eb9d76836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1436dd3df7284f9bb379236eb9d76836 already in progress: creating tablet
W20240503 02:12:27.072264 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3aad7d6dcf2e4009872ce156c536f38e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aad7d6dcf2e4009872ce156c536f38e already in progress: creating tablet
W20240503 02:12:27.072607 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 043ba14eaa5b46319b0b88457d2a5f45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 043ba14eaa5b46319b0b88457d2a5f45 already in progress: creating tablet
W20240503 02:12:27.073060 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2ecfd29eddde4c39b7661b1b7d96a08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ecfd29eddde4c39b7661b1b7d96a08e already in progress: creating tablet
W20240503 02:12:27.073388 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3f9994ca5aea48d3bc25c097dddba3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f9994ca5aea48d3bc25c097dddba3c3 already in progress: creating tablet
I20240503 02:12:27.078995 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_39060d5d029f4211a2c27b981453e636" } modify_external_catalogs: true
I20240503 02:12:27.079121 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_88f4791ae6054ec5becf0d13aec8be47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.079231 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_39060d5d029f4211a2c27b981453e636" } modify_external_catalogs: true
W20240503 02:12:27.079900 4274 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 62) took 1003 ms (client timeout 1000 ms). Trace:
W20240503 02:12:27.079895 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88f4791ae6054ec5becf0d13aec8be47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.080080 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_82701b51f62944de9442411ee0b446fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.080638 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82701b51f62944de9442411ee0b446fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.081600 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c4de876e162a48c1bba5469a0c9d6a34" } new_table_name: "default.Table_5920bfd139224d908c23f21d6a65308a" modify_external_catalogs: true
I20240503 02:12:27.084537 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3b9cdd4b8f9e4449aed0a52093a299b8" } modify_external_catalogs: true
I20240503 02:12:27.084792 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3b9cdd4b8f9e4449aed0a52093a299b8" } modify_external_catalogs: true
I20240503 02:12:27.090447 5286 raft_consensus.cc:482] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.090763 5286 raft_consensus.cc:504] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.092419 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:27.093272 4462 leader_election.cc:341] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:27.093168 4274 rpcz_store.cc:269] 0503 02:12:26.076246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.090865 (+ 14619us) service_pool.cc:225] Handling call
0503 02:12:27.079822 (+988957us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:26.905883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:26.905969 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:26.905974 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:26.905976 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:26.906065 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:26.906073 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:26.906078 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:26.906133 (+ 55us) tablet.cc:699] Row locks acquired
0503 02:12:26.906135 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:26.906162 (+ 27us) write_op.cc:270] Start()
0503 02:12:26.906186 (+ 24us) write_op.cc:276] Timestamp: P: 1714702346906116 usec, L: 0
0503 02:12:26.906189 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:26.906324 (+ 135us) log.cc:842] Serialized 527 byte log entry
0503 02:12:27.076900 (+170576us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.076992 (+ 92us) write_op.cc:301] APPLY: starting
0503 02:12:27.077004 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.077035 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.077069 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.077113 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.077141 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:27.077150 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:27.077213 (+ 63us) log.cc:842] Serialized 23 byte log entry
0503 02:12:27.077263 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.077282 (+ 19us) write_op.cc:454] Released schema lock
0503 02:12:27.077288 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:27.077341 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.077344 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":32,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":391,"num_ops":1,"prepare.queue_time_us":621953,"prepare.run_cpu_time_us":503,"prepare.run_wall_time_us":502,"replication_time_us":170693}]]}
I20240503 02:12:27.094332 5286 leader_election.cc:288] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.094403 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.096103 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4101d1a1a4c14c018fecdd3b3d05641f" } new_table_name: "default.Table_f24dcaf9ccd649c9ad2afc1764f3e6d5" modify_external_catalogs: true
I20240503 02:12:27.099963 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.100100 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.100268 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.100325 5114 tablet_replica.cc:330] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:27.100677 5114 raft_consensus.cc:2229] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:27.101019 5114 pending_rounds.cc:62] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:27.101106 5114 pending_rounds.cc:68] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420811332501504 op_type: NO_OP noop_request { }
I20240503 02:12:27.101263 5114 raft_consensus.cc:2866] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:27.101400 5114 raft_consensus.cc:2258] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.102737 5114 ts_tablet_manager.cc:1875] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:27.109093 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.111869 5159 ts_tablet_manager.cc:1888] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:27.122248 5106 raft_consensus.cc:348] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.122742 5106 raft_consensus.cc:374] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.130250 5159 log.cc:1196] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/959d998ec4ec4a52a6031650f2da4691
I20240503 02:12:27.137054 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a51563396614c54b980f0679e84a2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:18 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:27.151855 4649 consensus_meta.cc:287] T 8241a88f5c3a47578c92343f9fb5f7cd P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.649s user 0.005s sys 0.000s
I20240503 02:12:27.153136 5114 ts_tablet_manager.cc:1888] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:27.153710 5114 log.cc:1196] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/acdefb33fe6e4a9893dd66f6252395d7
I20240503 02:12:27.154682 5114 ts_tablet_manager.cc:1909] T acdefb33fe6e4a9893dd66f6252395d7 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:27.155062 4649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 409) took 1089 ms. Trace:
I20240503 02:12:27.155275 4649 rpcz_store.cc:276] 0503 02:12:26.062661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.062779 (+ 118us) service_pool.cc:225] Handling call
0503 02:12:26.115284 (+ 52505us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:26.115315 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:26.502774 (+387459us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.152222 (+649448us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:27.152515 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:27.156669 5106 raft_consensus.cc:729] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.157645 5106 consensus_queue.cc:260] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.168407 5598 raft_consensus.cc:482] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.168668 5598 raft_consensus.cc:504] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.169623 5598 leader_election.cc:288] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.138724 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:27.174675 5611 raft_consensus.cc:482] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:27.175180 4462 leader_election.cc:341] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:27.175387 4462 leader_election.cc:302] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:27.175837 5589 raft_consensus.cc:2726] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:27.176888 5106 ts_tablet_manager.cc:1398] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.115s user 0.003s sys 0.000s
I20240503 02:12:27.180320 5611 raft_consensus.cc:504] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.181509 5159 ts_tablet_manager.cc:1909] T 959d998ec4ec4a52a6031650f2da4691 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:27.182009 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 387) took 2121 ms. Trace:
I20240503 02:12:27.182170 5159 rpcz_store.cc:276] 0503 02:12:25.060224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.060512 (+ 288us) service_pool.cc:225] Handling call
0503 02:12:26.373681 (+1313169us) ts_tablet_manager.cc:1120] Deleting tablet 959d998ec4ec4a52a6031650f2da4691
0503 02:12:26.373692 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.961001 (+587309us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:26.971733 (+ 10732us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.181952 (+210219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1312914}
I20240503 02:12:27.182817 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 388) took 2122 ms. Trace:
I20240503 02:12:27.182956 5159 rpcz_store.cc:276] 0503 02:12:25.060513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.060805 (+ 292us) service_pool.cc:225] Handling call
0503 02:12:27.182617 (+2121812us) ts_tablet_manager.cc:1120] Deleting tablet 4b4972057fae45168756d7c28ddbe2bf
0503 02:12:27.182631 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.182770 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121567}
I20240503 02:12:27.183259 5159 tablet_replica.cc:330] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:27.183553 5159 raft_consensus.cc:2229] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:27.183802 5159 raft_consensus.cc:2258] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.184525 5159 ts_tablet_manager.cc:1875] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.186539 5611 leader_election.cc:288] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:27.187909 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:27.188159 4529 raft_consensus.cc:2454] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:27.187835 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:27.189602 4813 raft_consensus.cc:2454] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:27.191215 5106 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:27.196424 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3e1aa615a80d47258acd2c493d6cb299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.197064 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e1aa615a80d47258acd2c493d6cb299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.208470 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 959d998ec4ec4a52a6031650f2da4691 (table default.Table_f556497cf91b4b6cb71095aceb0c3a58 [id=d85f957a661b45e081583db26de0b03b]) successfully deleted
W20240503 02:12:27.209244 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4b4972057fae45168756d7c28ddbe2bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4972057fae45168756d7c28ddbe2bf already in progress: creating tablet
I20240503 02:12:27.222250 5106 tablet_bootstrap.cc:654] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.224556 5114 tablet_replica.cc:330] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:27.225009 5114 raft_consensus.cc:2229] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:27.225531 5114 raft_consensus.cc:2258] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.222818 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:27.226006 5589 raft_consensus.cc:482] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.226557 5589 raft_consensus.cc:504] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.227286 4674 raft_consensus.cc:2454] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:27.229514 5589 leader_election.cc:288] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.232601 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet acdefb33fe6e4a9893dd66f6252395d7 (table default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]) successfully deleted
I20240503 02:12:27.234848 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:27.236433 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:27.237063 4635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 89) took 1016 ms. Trace:
I20240503 02:12:27.237560 4674 raft_consensus.cc:2454] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:27.237946 4635 rpcz_store.cc:276] 0503 02:12:26.219851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.219974 (+ 123us) service_pool.cc:225] Handling call
0503 02:12:26.488677 (+268703us) spinlock_profiling.cc:243] Waited 7.73 ms on lock 0x616000122854. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.170880 (+682203us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x616000142054. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.236142 (+ 65262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":564307456}
I20240503 02:12:27.238323 4462 leader_election.cc:302] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:27.240159 5286 raft_consensus.cc:2781] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.241539 5286 raft_consensus.cc:482] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.242551 5286 raft_consensus.cc:3037] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.245379 4671 raft_consensus.cc:2454] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:27.246043 4462 leader_election.cc:302] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:27.246721 5589 raft_consensus.cc:2781] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.246898 5589 raft_consensus.cc:482] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.247062 5589 raft_consensus.cc:3037] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.247189 5106 tablet_bootstrap.cc:492] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:27.247540 5106 ts_tablet_manager.cc:1367] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.006s
I20240503 02:12:27.249017 5106 raft_consensus.cc:348] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.250113 5589 raft_consensus.cc:504] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.194887 4746 leader_election.cc:302] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:27.254997 5103 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:27.255208 5103 ts_tablet_manager.cc:1367] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.297s user 0.026s sys 0.053s
I20240503 02:12:27.256278 5103 raft_consensus.cc:348] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.256637 5103 raft_consensus.cc:374] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.256760 5103 raft_consensus.cc:729] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:27.259335 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 64647.8us
W20240503 02:12:27.267462 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 868a6ad6cc1a40f091b4516676c24c97 has no consensus state
W20240503 02:12:27.268218 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3db446d8a6984d9cb4175887f7e125bb consensus state has no leader
I20240503 02:12:27.272632 5611 raft_consensus.cc:2781] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.273945 5611 raft_consensus.cc:482] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.274128 5611 raft_consensus.cc:3037] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.277817 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:27.291622 4778 consensus_meta.cc:287] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.517s user 0.002s sys 0.002s
I20240503 02:12:27.292711 4811 raft_consensus.cc:2454] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:27.300859 5105 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:27.301446 5105 ts_tablet_manager.cc:1367] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.648s user 0.006s sys 0.006s
I20240503 02:12:27.302572 5105 raft_consensus.cc:348] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.303938 5105 raft_consensus.cc:374] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.304167 5105 raft_consensus.cc:729] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.304883 5611 raft_consensus.cc:504] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.306946 5105 consensus_queue.cc:260] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.309453 5611 leader_election.cc:288] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:27.309574 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 41ef3c6e5e7d40249b112144975ba220 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:27.309839 5105 ts_tablet_manager.cc:1398] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:27.310276 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:27.310596 4529 raft_consensus.cc:2379] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:27.310606 5105 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:27.311468 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet c0c18cecefab4b6caed0039d09cc1890 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:27.313787 5105 tablet_bootstrap.cc:654] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.314586 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5916ed7fb62a4cdc910846e24178b911" } new_table_name: "default.Table_9313c394ecf4460a81267e309fdd6748" modify_external_catalogs: true
I20240503 02:12:27.317472 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ab30b57f7d9044fca1bf76dc5e2bc323" } modify_external_catalogs: true
I20240503 02:12:27.317696 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ab30b57f7d9044fca1bf76dc5e2bc323" } modify_external_catalogs: true
I20240503 02:12:27.318176 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2243856948a74aaab070be52ee0905a2" } new_table_name: "default.Table_001ef146e7b442e99269b86b5b32c214" modify_external_catalogs: true
I20240503 02:12:27.318877 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_174379bd4f48483fa1baddfe455090de" } new_table_name: "default.Table_5e542f394cce413f99f8012fcc9fba9a" modify_external_catalogs: true
I20240503 02:12:27.326303 5589 leader_election.cc:288] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.326731 5589 consensus_queue.cc:1035] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.330771 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:27.331128 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:27.331331 4275 catalog_manager.cc:5484] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:27.331792 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]). Expected version 3 got 0
I20240503 02:12:27.332180 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:27.332566 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:27.334079 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa25fee059b46ec915bcc1a01cb8b27 (DEFAULT_TABLE table=default.Table_d6eeea17720743c8b78544cd9e6efefd [id=e68bbe8a4fd14c1a86f5b81cf8bc06e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.334600 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa25fee059b46ec915bcc1a01cb8b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.335806 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7ca2fea26749cfa4e4213792c778ce (DEFAULT_TABLE table=default.Table_1de4326d5bda4d70afc8b879ab9acb6f [id=867d8add592f498293ded30a841aa893]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.336333 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7ca2fea26749cfa4e4213792c778ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.338148 5103 consensus_queue.cc:260] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.339112 5103 ts_tablet_manager.cc:1398] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.084s user 0.001s sys 0.001s
I20240503 02:12:27.345536 5103 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:27.347131 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0745a55223475893edd01d904a223b (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.347518 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 9df090ed6179485eb6922240e4c11ad1 (DEFAULT_TABLE table=default.Table_1e24815702a84aa281e889334d8b734f [id=143cfb566db2475aa4163d4a17edb8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.348055 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df090ed6179485eb6922240e4c11ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.350695 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 23ccd25074db40fb96053b915125a425 (DEFAULT_TABLE table=default.Table_8eb9b75d260a42229a49d6343e0b2e3a [id=0345d41c213e43ef875c7d3e898b3fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.351275 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ccd25074db40fb96053b915125a425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.354581 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0745a55223475893edd01d904a223b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:27.359406 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 368aabce19984d03aa7116c8df990ef1 consensus state has no leader
I20240503 02:12:27.360220 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3b72c20a4b5a4d2ea266f031733d1aab
I20240503 02:12:27.360798 4270 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 609) took 1072 ms. Trace:
I20240503 02:12:27.360951 4270 rpcz_store.cc:276] 0503 02:12:26.288594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.066786 (+778192us) service_pool.cc:225] Handling call
0503 02:12:27.078887 (+ 12101us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:27.079344 (+ 457us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:27.079486 (+ 142us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:27.079546 (+ 60us) catalog_manager.cc:2758] Locking tablets
0503 02:12:27.328903 (+249357us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:27.360018 (+ 31115us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:27.360107 (+ 89us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:27.360151 (+ 44us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:27.360754 (+ 603us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.333068 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:27.333192 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:27.333198 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:27.333201 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:27.333329 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:27.333342 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:27.333347 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:27.333461 (+ 114us) tablet.cc:699] Row locks acquired
0503 02:12:27.333464 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:27.333502 (+ 38us) write_op.cc:270] Start()
0503 02:12:27.333536 (+ 34us) write_op.cc:276] Timestamp: P: 1714702347333463 usec, L: 0
0503 02:12:27.333540 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.333757 (+ 217us) log.cc:842] Serialized 725 byte log entry
0503 02:12:27.356982 (+ 23225us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.358584 (+ 1602us) write_op.cc:301] APPLY: starting
0503 02:12:27.358595 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.358623 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.358651 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.358792 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.358821 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:27.358829 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:27.358880 (+ 51us) log.cc:842] Serialized 29 byte log entry
0503 02:12:27.358920 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.358935 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:27.358940 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:27.358987 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.358991 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":27157376,"child_traces":[["op",{"apply.queue_time_us":1565,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":453,"num_ops":2,"prepare.queue_time_us":3309,"prepare.run_cpu_time_us":771,"prepare.run_wall_time_us":771,"replication_time_us":23416,"threads_started":1}]]}
I20240503 02:12:27.362128 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:27.363929 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9470480a9de45209044f8e8c461519a consensus state has no leader
I20240503 02:12:27.365216 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 43fc277fa4884c7f92118e333226e800 (DEFAULT_TABLE table=default.Table_466756ec40864212abe6bd1a32875b14 [id=c3fac625561b486c9052e6cf111c2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.365754 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fc277fa4884c7f92118e333226e800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.347676 5103 tablet_bootstrap.cc:654] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.368376 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 23ccd25074db40fb96053b915125a425 (DEFAULT_TABLE table=default.Table_8eb9b75d260a42229a49d6343e0b2e3a [id=0345d41c213e43ef875c7d3e898b3fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.368805 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ccd25074db40fb96053b915125a425. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:27.369709 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b143c398ccbe42f09fa2e33f04f0d4d7 consensus state has no leader
I20240503 02:12:27.370196 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.370390 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 9df090ed6179485eb6922240e4c11ad1 (DEFAULT_TABLE table=default.Table_1e24815702a84aa281e889334d8b734f [id=143cfb566db2475aa4163d4a17edb8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.370914 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df090ed6179485eb6922240e4c11ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.371958 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0745a55223475893edd01d904a223b (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.372781 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0745a55223475893edd01d904a223b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.372565 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa25fee059b46ec915bcc1a01cb8b27 (DEFAULT_TABLE table=default.Table_d6eeea17720743c8b78544cd9e6efefd [id=e68bbe8a4fd14c1a86f5b81cf8bc06e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.374814 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa25fee059b46ec915bcc1a01cb8b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.375936 5103 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:27.376183 5103 ts_tablet_manager.cc:1367] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240503 02:12:27.376408 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7ca2fea26749cfa4e4213792c778ce (DEFAULT_TABLE table=default.Table_1de4326d5bda4d70afc8b879ab9acb6f [id=867d8add592f498293ded30a841aa893]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.378099 5103 raft_consensus.cc:348] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.378440 5103 raft_consensus.cc:374] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.378528 5103 raft_consensus.cc:729] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.378824 5103 consensus_queue.cc:260] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.379403 5103 ts_tablet_manager.cc:1398] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:27.379766 5103 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:27.385859 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7ca2fea26749cfa4e4213792c778ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.389425 5103 tablet_bootstrap.cc:654] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.397015 5103 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:27.397217 5103 ts_tablet_manager.cc:1367] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 02:12:27.398944 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 23ccd25074db40fb96053b915125a425
I20240503 02:12:27.399319 5103 raft_consensus.cc:348] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.399536 5103 raft_consensus.cc:374] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.399636 4268 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 606) took 1173 ms. Trace:
I20240503 02:12:27.399669 5103 raft_consensus.cc:729] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.399921 5103 consensus_queue.cc:260] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.399746 4268 rpcz_store.cc:276] 0503 02:12:26.225777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.895036 (+669259us) service_pool.cc:225] Handling call
0503 02:12:26.908543 (+ 13507us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:26.909142 (+ 599us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:26.909355 (+ 213us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:26.909451 (+ 96us) catalog_manager.cc:2758] Locking tablets
0503 02:12:27.329176 (+419725us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:27.398736 (+ 69560us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:27.398837 (+ 101us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:27.398885 (+ 48us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:27.399540 (+ 655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.333868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:27.333961 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:27.333966 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:27.333968 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:27.334056 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:27.334068 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:27.334073 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:27.334215 (+ 142us) tablet.cc:699] Row locks acquired
0503 02:12:27.334218 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:27.334253 (+ 35us) write_op.cc:270] Start()
0503 02:12:27.334283 (+ 30us) write_op.cc:276] Timestamp: P: 1714702347334214 usec, L: 0
0503 02:12:27.334286 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.334454 (+ 168us) log.cc:842] Serialized 725 byte log entry
0503 02:12:27.397311 (+ 62857us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.397587 (+ 276us) write_op.cc:301] APPLY: starting
0503 02:12:27.397614 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.397684 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.397738 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.397845 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.398072 (+ 227us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:27.398085 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:27.398257 (+ 172us) log.cc:842] Serialized 29 byte log entry
0503 02:12:27.398609 (+ 352us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.398638 (+ 29us) write_op.cc:454] Released schema lock
0503 02:12:27.398647 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":30375808,"child_traces":[["op",{"apply.queue_time_us":215,"num_ops":2,"prepare.queue_time_us":4055,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":640,"replication_time_us":62997}]]}
I20240503 02:12:27.400465 5103 ts_tablet_manager.cc:1398] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 02:12:27.400748 5103 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:27.402693 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d39f042afb954daa923502ef12377bd5
I20240503 02:12:27.403205 4266 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 607) took 1173 ms. Trace:
I20240503 02:12:27.403360 4266 rpcz_store.cc:276] 0503 02:12:26.229817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.059863 (+830046us) service_pool.cc:225] Handling call
0503 02:12:27.065634 (+ 5771us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:27.066153 (+ 519us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:27.066302 (+ 149us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:27.066376 (+ 74us) catalog_manager.cc:2758] Locking tablets
0503 02:12:27.329528 (+263152us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:27.402476 (+ 72948us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:27.402564 (+ 88us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:27.402611 (+ 47us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:27.403162 (+ 551us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.334546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:27.334639 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:27.334646 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:27.334648 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:27.334729 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:27.334737 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:27.334741 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:27.334823 (+ 82us) tablet.cc:699] Row locks acquired
0503 02:12:27.334825 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:27.334850 (+ 25us) write_op.cc:270] Start()
0503 02:12:27.334877 (+ 27us) write_op.cc:276] Timestamp: P: 1714702347334815 usec, L: 0
0503 02:12:27.334880 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.335009 (+ 129us) log.cc:842] Serialized 756 byte log entry
0503 02:12:27.397398 (+ 62389us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.401804 (+ 4406us) write_op.cc:301] APPLY: starting
0503 02:12:27.401819 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.401868 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.401905 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.401988 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.402024 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:27.402032 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:27.402116 (+ 84us) log.cc:842] Serialized 29 byte log entry
0503 02:12:27.402361 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.402391 (+ 30us) write_op.cc:454] Released schema lock
0503 02:12:27.402398 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":12347776,"child_traces":[["op",{"apply.queue_time_us":4370,"num_ops":2,"prepare.queue_time_us":4467,"prepare.run_cpu_time_us":510,"prepare.run_wall_time_us":509,"replication_time_us":62499}]]}
I20240503 02:12:27.407050 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8241a88f5c3a47578c92343f9fb5f7cd
I20240503 02:12:27.407788 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6623c5960394449394b0bc3dae42bdd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.407896 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6d6c3b05fef34441af532f58a3fb5f6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.408440 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d6c3b05fef34441af532f58a3fb5f6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.409482 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_186402d8f9e2484c89977667d29d1ff9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.409890 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6623c5960394449394b0bc3dae42bdd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:27.409976 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_186402d8f9e2484c89977667d29d1ff9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.411026 5103 tablet_bootstrap.cc:654] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.412406 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8ce69a229b7b49b5b712f243d5d4fbb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.413100 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.414661 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a3621f453c4d4329acee2b8d139c662a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.415307 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3621f453c4d4329acee2b8d139c662a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.416707 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c269ebd3c64a4b61964defe4593937ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.417168 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c269ebd3c64a4b61964defe4593937ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.418907 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_23f94387862b4b6784389cd83a9ac77e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.419412 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23f94387862b4b6784389cd83a9ac77e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.420802 5115 ts_tablet_manager.cc:1888] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:27.427043 5106 raft_consensus.cc:374] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.429071 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:27.431982 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ccd25074db40fb96053b915125a425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:26 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.432354 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.432511 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.432780 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ccd25074db40fb96053b915125a425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:26 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.433948 5159 ts_tablet_manager.cc:1888] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:27.434135 5159 log.cc:1196] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
I20240503 02:12:27.434080 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:27.434298 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]). Expected version 3 got 0
I20240503 02:12:27.434537 5159 ts_tablet_manager.cc:1909] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:27.434756 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:27.434876 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 389) took 2373 ms. Trace:
I20240503 02:12:27.434970 5159 rpcz_store.cc:276] 0503 02:12:25.060906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.060975 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:27.183183 (+2122208us) ts_tablet_manager.cc:1120] Deleting tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
0503 02:12:27.183191 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.310311 (+127120us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.328119 (+ 17808us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.434824 (+106705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121997}
I20240503 02:12:27.435065 4273 catalog_manager.cc:5484] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:27.435523 5159 tablet_replica.cc:330] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:27.435603 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:27.435735 5159 raft_consensus.cc:2229] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:27.435904 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:27.435937 5159 raft_consensus.cc:2258] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.436185 5105 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:27.436183 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_90d95bd149e04a83b065dba3bf38e43e [id=5108d90cc915403ba879db13f6a9f1be]). Expected version 1 got 0
I20240503 02:12:27.436442 5105 ts_tablet_manager.cc:1367] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.001s
I20240503 02:12:27.436623 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c0b9003f248b47a9952e6f995887ae0c
I20240503 02:12:27.437122 4267 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 608) took 1195 ms. Trace:
I20240503 02:12:27.437156 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.437230 4267 rpcz_store.cc:276] 0503 02:12:26.242092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.060734 (+818642us) service_pool.cc:225] Handling call
0503 02:12:27.084450 (+ 23716us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:27.085002 (+ 552us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:27.085179 (+ 177us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:27.085310 (+ 131us) catalog_manager.cc:2758] Locking tablets
0503 02:12:27.085480 (+ 170us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:27.268090 (+182610us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:27.268216 (+ 126us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:27.436576 (+168360us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:27.437068 (+ 492us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.203051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:27.203226 (+ 175us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:27.203231 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:27.203233 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:27.203497 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:27.203513 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:27.203549 (+ 36us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:27.204108 (+ 559us) tablet.cc:699] Row locks acquired
0503 02:12:27.204111 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:27.204167 (+ 56us) write_op.cc:270] Start()
0503 02:12:27.204222 (+ 55us) write_op.cc:276] Timestamp: P: 1714702347204140 usec, L: 0
0503 02:12:27.204226 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.204982 (+ 756us) log.cc:842] Serialized 790 byte log entry
0503 02:12:27.266300 (+ 61318us) op_driver.cc:444] REPLICATION: finished
0503 02:12:27.267216 (+ 916us) write_op.cc:301] APPLY: starting
0503 02:12:27.267228 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:27.267261 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:27.267293 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:27.267354 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:27.267380 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:27.267387 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:27.267452 (+ 65us) log.cc:842] Serialized 29 byte log entry
0503 02:12:27.267495 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.267508 (+ 13us) write_op.cc:454] Released schema lock
0503 02:12:27.267513 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:27.267577 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:27.267581 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":53921536,"child_traces":[["op",{"apply.queue_time_us":880,"apply.run_cpu_time_us":412,"apply.run_wall_time_us":411,"num_ops":2,"prepare.queue_time_us":116599,"prepare.run_cpu_time_us":977,"prepare.run_wall_time_us":2209,"replication_time_us":62056}]]}
I20240503 02:12:27.438517 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_023bb9a5caa9417d9ab87c83b288de66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.439110 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_023bb9a5caa9417d9ab87c83b288de66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.441907 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (table default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]) successfully deleted
I20240503 02:12:27.441848 5105 raft_consensus.cc:348] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.442237 5105 raft_consensus.cc:374] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.442381 5105 raft_consensus.cc:729] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.442811 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b9003f248b47a9952e6f995887ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.442862 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b9003f248b47a9952e6f995887ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.443204 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.443244 5105 consensus_queue.cc:260] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.443519 4674 raft_consensus.cc:3037] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.443547 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa25fee059b46ec915bcc1a01cb8b27 (DEFAULT_TABLE table=default.Table_d6eeea17720743c8b78544cd9e6efefd [id=e68bbe8a4fd14c1a86f5b81cf8bc06e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.444033 5105 ts_tablet_manager.cc:1398] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:27.444381 5105 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:27.445106 5286 raft_consensus.cc:504] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.445941 5286 leader_election.cc:288] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.446799 4674 raft_consensus.cc:2454] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:27.447201 4746 leader_election.cc:302] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:27.447516 5611 raft_consensus.cc:2781] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.447796 5611 raft_consensus.cc:686] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:27.447947 4632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 91) took 1161 ms. Trace:
I20240503 02:12:27.448057 4632 rpcz_store.cc:276] 0503 02:12:26.286759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.287021 (+ 262us) service_pool.cc:225] Handling call
0503 02:12:26.484993 (+197972us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x616000122854. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.178418 (+693425us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x616000142054. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.447892 (+269474us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":521652736}
I20240503 02:12:27.448153 5611 consensus_queue.cc:237] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.452136 5105 tablet_bootstrap.cc:654] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.436450 5159 ts_tablet_manager.cc:1875] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.455603 5637 consensus_queue.cc:1035] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.456357 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:27.457088 5106 raft_consensus.cc:729] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.461251 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa25fee059b46ec915bcc1a01cb8b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.463024 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet aa67857b858c4719b498291c52b7aceb from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:27.471859 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e42fa6e49c394853b67735ce31f70017" } new_table_name: "default.Table_0cc3309db13041969f6cbdf461a53f8f" modify_external_catalogs: true
I20240503 02:12:27.477097 5115 log.cc:1196] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
I20240503 02:12:27.478816 5114 ts_tablet_manager.cc:1875] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.478964 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7ca2fea26749cfa4e4213792c778ce (DEFAULT_TABLE table=default.Table_1de4326d5bda4d70afc8b879ab9acb6f [id=867d8add592f498293ded30a841aa893]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.479743 5106 consensus_queue.cc:260] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.480199 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0745a55223475893edd01d904a223b (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.480751 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 23ccd25074db40fb96053b915125a425 (DEFAULT_TABLE table=default.Table_8eb9b75d260a42229a49d6343e0b2e3a [id=0345d41c213e43ef875c7d3e898b3fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.481308 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7ca2fea26749cfa4e4213792c778ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.482517 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ccd25074db40fb96053b915125a425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.483719 5106 ts_tablet_manager.cc:1398] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.233s user 0.002s sys 0.000s
I20240503 02:12:27.484902 4671 raft_consensus.cc:2379] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
W20240503 02:12:27.485288 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 177838us
I20240503 02:12:27.485949 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a54873b48847c39a2b961ab5c3806c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:27.492274 5106 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:27.492440 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 9df090ed6179485eb6922240e4c11ad1 (DEFAULT_TABLE table=default.Table_1e24815702a84aa281e889334d8b734f [id=143cfb566db2475aa4163d4a17edb8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.493181 5115 ts_tablet_manager.cc:1909] T 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:27.495793 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 6624e145aac2412d98acf847e9e9c8b8 (DEFAULT_TABLE table=default.Table_ae1fa81b7e334c958fc7d781ecd7bc11 [id=340f7dfafe334dbdbed043aab0b2a97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.496006 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 43fc277fa4884c7f92118e333226e800 (DEFAULT_TABLE table=default.Table_466756ec40864212abe6bd1a32875b14 [id=c3fac625561b486c9052e6cf111c2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.496466 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fc277fa4884c7f92118e333226e800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.496565 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624e145aac2412d98acf847e9e9c8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.499058 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 6624e145aac2412d98acf847e9e9c8b8 (DEFAULT_TABLE table=default.Table_ae1fa81b7e334c958fc7d781ecd7bc11 [id=340f7dfafe334dbdbed043aab0b2a97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.499631 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624e145aac2412d98acf847e9e9c8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.499631 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9a928f9047e94439b5d83771a9629ffe" } new_table_name: "default.Table_612f648735a34eff8dafc63f119da941" modify_external_catalogs: true
I20240503 02:12:27.500962 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 (DEFAULT_TABLE table=default.Table_301d83f516e84342a226f385300cef79 [id=8d41860986c04f2b9b365c0af0744299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.501543 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af50eb2b1c949a0a6a71e64a894e8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.504091 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:27.504429 4812 raft_consensus.cc:3037] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.507447 4812 raft_consensus.cc:2454] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:27.511682 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8ac84b01b9b94262b9e66dcfb42ded1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.512310 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac84b01b9b94262b9e66dcfb42ded1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.518919 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9b6b0e4d8d9f43bf9322382e80b8e842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.519652 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b6b0e4d8d9f43bf9322382e80b8e842 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.526554 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 38614de3642148aa82e5108d9160f537 (DEFAULT_TABLE table=default.Table_7dbf326ce1e445b2a3d9f194413f2196 [id=bf87915d1c8840a1893377bd5d12a482]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.527016 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38614de3642148aa82e5108d9160f537. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.530777 5114 ts_tablet_manager.cc:1888] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:27.533736 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 38614de3642148aa82e5108d9160f537 (DEFAULT_TABLE table=default.Table_7dbf326ce1e445b2a3d9f194413f2196 [id=bf87915d1c8840a1893377bd5d12a482]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.534248 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd (DEFAULT_TABLE table=default.Table_47b984d6c813457786ac8b9c4fad0255 [id=9ab5a5bf648e486a97d8ea223c90dc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.534651 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a915d760ba14a31895a5dc7ec5b34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.534938 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd (DEFAULT_TABLE table=default.Table_47b984d6c813457786ac8b9c4fad0255 [id=9ab5a5bf648e486a97d8ea223c90dc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.535362 4496 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 (DEFAULT_TABLE table=default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01 [id=2be6fa081ef04895871c5260f53f0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.535712 4498 tablet_service.cc:1467] Processing CreateTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.536275 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 17a24b5c83b649e79644b08253d851d6 (DEFAULT_TABLE table=default.Table_7fe2fd4286f4411aad562211e4bd77ae [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.536631 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a24b5c83b649e79644b08253d851d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.537194 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 (DEFAULT_TABLE table=default.Table_5920bfd139224d908c23f21d6a65308a [id=6c79ef76b01744f0a83455d15c08d34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.537427 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 17a24b5c83b649e79644b08253d851d6 (DEFAULT_TABLE table=default.Table_7fe2fd4286f4411aad562211e4bd77ae [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.537700 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 (DEFAULT_TABLE table=default.Table_5920bfd139224d908c23f21d6a65308a [id=6c79ef76b01744f0a83455d15c08d34a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:27.537814 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_02a490be3c6544d5a09aa2551909cee3 [id=fde711a2c8e04377996a7cd87f629e68], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7efef931ca554ae18579bbea8d1c04f7 consensus state has no leader
I20240503 02:12:27.538225 4791 tablet_service.cc:1467] Processing CreateTablet for tablet a6de1b96c15c4b44809a03ef99e077ba (DEFAULT_TABLE table=default.Table_98865c1dc735482d9fdb50d75f612ac9 [id=f40c534a8761468f970216fb0260c33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.537204 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.538769 4780 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 (DEFAULT_TABLE table=default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01 [id=2be6fa081ef04895871c5260f53f0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.545120 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868a6ad6cc1a40f091b4516676c24c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.549108 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd27023b3f47f281dfb5474ea3a4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.549691 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6de1b96c15c4b44809a03ef99e077ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.551836 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c919bda1eaaa4e52828aac3ef337f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.554416 4462 leader_election.cc:302] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:27.554802 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 44991.8us
I20240503 02:12:27.555508 5598 raft_consensus.cc:482] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.555821 4631 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 92) took 1195 ms. Trace:
I20240503 02:12:27.559432 4784 tablet_service.cc:1467] Processing CreateTablet for tablet d7aaccb55c33483dbb61d847418e831e (DEFAULT_TABLE table=default.Table_a5a36396c8974489a65a6510c520cee7 [id=b96ca4d66c3d444f99478e250aa30925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.559954 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aaccb55c33483dbb61d847418e831e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.562732 4631 rpcz_store.cc:276] 0503 02:12:26.286971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.287042 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:27.159705 (+872663us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x616000122e54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.482737 (+323032us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418646784}
I20240503 02:12:27.563129 5382 raft_consensus.cc:482] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.563299 5382 raft_consensus.cc:504] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.563975 5382 leader_election.cc:288] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:27.564023 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 (DEFAULT_TABLE table=default.Table_1aea91e895ce4610be1ebf9ee910958b [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.566035 4633 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 90) took 1336 ms. Trace:
I20240503 02:12:27.566143 4633 rpcz_store.cc:276] 0503 02:12:26.222440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.222533 (+ 93us) service_pool.cc:225] Handling call
0503 02:12:27.159775 (+937242us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x616000122e54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:27.559058 (+399283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665227392}
I20240503 02:12:27.567508 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0745a55223475893edd01d904a223b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.568372 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 43fc277fa4884c7f92118e333226e800 (DEFAULT_TABLE table=default.Table_466756ec40864212abe6bd1a32875b14 [id=c3fac625561b486c9052e6cf111c2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.570168 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df090ed6179485eb6922240e4c11ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.570902 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79884b3e73904c708d0d48bca85351b7" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:27.573036 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.575065 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8cdd736fadb64d9faf9eb743916fa0b8" } new_table_name: "default.Table_3295e34cbed747c68141113383a688b1" modify_external_catalogs: true
I20240503 02:12:27.576530 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6e29e77c800340dbad41e6a6b3c1c0ec" } new_table_name: "default.Table_bb051703036147bd9a71497d00edc307" modify_external_catalogs: true
I20240503 02:12:27.577400 4670 raft_consensus.cc:3037] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.577855 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_301d83f516e84342a226f385300cef79" } modify_external_catalogs: true
I20240503 02:12:27.578040 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_301d83f516e84342a226f385300cef79" } modify_external_catalogs: true
W20240503 02:12:27.581137 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cc2ebfb11aa4840adcc35da73be75a2 consensus state has no leader
I20240503 02:12:27.582727 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b9003f248b47a9952e6f995887ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.583043 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.583556 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fc277fa4884c7f92118e333226e800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.585059 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ccd25074db40fb96053b915125a425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:26 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.585039 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_90d95bd149e04a83b065dba3bf38e43e" } new_table_name: "default.Table_77eae5a54c4c440a9928ae87553755be" modify_external_catalogs: true
I20240503 02:12:27.585637 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.586347 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.590193 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 41d44fe8f6334572996665bdec60e2e5 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:27.594193 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.593921 5637 consensus_queue.cc:1035] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240503 02:12:27.594429 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bb9ac773c4bc685cb8193e574460b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.595701 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.598574 4643 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 (DEFAULT_TABLE table=default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01 [id=2be6fa081ef04895871c5260f53f0a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.600646 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 38614de3642148aa82e5108d9160f537 (DEFAULT_TABLE table=default.Table_7dbf326ce1e445b2a3d9f194413f2196 [id=bf87915d1c8840a1893377bd5d12a482]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.603412 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.604578 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_93262e3fdea24977a7a3aa557373b9eb" } modify_external_catalogs: true
I20240503 02:12:27.604802 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_93262e3fdea24977a7a3aa557373b9eb" } modify_external_catalogs: true
I20240503 02:12:27.607230 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:27.608368 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_033e1f689d894383a2e2c0ebbbc6023b" } modify_external_catalogs: true
I20240503 02:12:27.608649 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_033e1f689d894383a2e2c0ebbbc6023b" } modify_external_catalogs: true
W20240503 02:12:27.608763 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee1fac803dd84eee8722f0a14559faf6 consensus state has no leader
I20240503 02:12:27.609150 5115 tablet_replica.cc:330] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:27.609433 5115 raft_consensus.cc:2229] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:27.609731 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f8b6599206549dd8ddf449a52c61fce consensus state has no leader
I20240503 02:12:27.611917 5598 raft_consensus.cc:504] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.613458 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b8df46519f1a401b862bbfabff3577fe" } modify_external_catalogs: true
I20240503 02:12:27.613745 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b8df46519f1a401b862bbfabff3577fe" } modify_external_catalogs: true
I20240503 02:12:27.614462 5115 raft_consensus.cc:2258] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.617256 4501 tablet_service.cc:1467] Processing CreateTablet for tablet a6de1b96c15c4b44809a03ef99e077ba (DEFAULT_TABLE table=default.Table_98865c1dc735482d9fdb50d75f612ac9 [id=f40c534a8761468f970216fb0260c33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.617976 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 (DEFAULT_TABLE table=default.Table_301d83f516e84342a226f385300cef79 [id=8d41860986c04f2b9b365c0af0744299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.620165 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 (DEFAULT_TABLE table=default.Table_1aea91e895ce4610be1ebf9ee910958b [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.618903 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 (DEFAULT_TABLE table=default.Table_1aea91e895ce4610be1ebf9ee910958b [id=5ca70228a70e4c02995e3c058a569e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.623556 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd27023b3f47f281dfb5474ea3a4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.625633 5115 ts_tablet_manager.cc:1875] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.626340 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38614de3642148aa82e5108d9160f537. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.627768 4490 tablet_service.cc:1467] Processing CreateTablet for tablet d7aaccb55c33483dbb61d847418e831e (DEFAULT_TABLE table=default.Table_a5a36396c8974489a65a6510c520cee7 [id=b96ca4d66c3d444f99478e250aa30925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.629081 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.631671 5589 raft_consensus.cc:2726] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:27.631894 5114 log.cc:1196] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:27.635452 4462 leader_election.cc:302] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:27.635746 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 80700.7us
I20240503 02:12:27.637732 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:27.637991 4809 raft_consensus.cc:2454] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:27.638464 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af50eb2b1c949a0a6a71e64a894e8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.640254 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.640298 4647 tablet_service.cc:1467] Processing CreateTablet for tablet d7aaccb55c33483dbb61d847418e831e (DEFAULT_TABLE table=default.Table_a5a36396c8974489a65a6510c520cee7 [id=b96ca4d66c3d444f99478e250aa30925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.641520 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aaccb55c33483dbb61d847418e831e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.646462 5105 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:27.647003 5105 ts_tablet_manager.cc:1367] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.203s user 0.006s sys 0.002s
I20240503 02:12:27.650373 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cc2e1a713cc449a68a25796a8b26205a" } modify_external_catalogs: true
I20240503 02:12:27.650911 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cc2e1a713cc449a68a25796a8b26205a" } modify_external_catalogs: true
I20240503 02:12:27.653298 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a (table default.Table_535d1456ce3640c6a7e33dbc4b2c305c [id=eae0ce8b19494370998ad5f3c565c7ee]) successfully deleted
W20240503 02:12:27.653652 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2a51563396614c54b980f0679e84a2b1 because the tablet was not found. No further retry: Not found: Tablet not found: 2a51563396614c54b980f0679e84a2b1
W20240503 02:12:27.653874 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 959d998ec4ec4a52a6031650f2da4691 because the tablet was not found. No further retry: Not found: Tablet not found: 959d998ec4ec4a52a6031650f2da4691
W20240503 02:12:27.654177 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3b72c20a4b5a4d2ea266f031733d1aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b72c20a4b5a4d2ea266f031733d1aab already in progress: creating tablet
I20240503 02:12:27.654349 5668 raft_consensus.cc:2781] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.654529 5668 raft_consensus.cc:686] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:27.655367 5652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:27.655679 5432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59500 (request call id 0) took 3045 ms. Trace:
I20240503 02:12:27.655747 5652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:27.656447 5598 leader_election.cc:288] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.657300 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:27.662801 5286 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.662840 5659 raft_consensus.cc:482] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.663024 5286 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.663058 5659 raft_consensus.cc:504] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.663833 5286 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.664299 5659 leader_election.cc:288] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:27.677337 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aaccb55c33483dbb61d847418e831e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.679689 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.680008 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c18cecefab4b6caed0039d09cc1890 with delete_type TABLET_DATA_DELETED (replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.680156 5114 ts_tablet_manager.cc:1909] T b28bb9ac773c4bc685cb8193e574460b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:27.680483 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.680691 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 79884b3e73904c708d0d48bca85351b7 with delete_type TABLET_DATA_DELETED (replaced by tablet cf10c48d165047668fbd5ca17f917fe3) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.681654 5105 raft_consensus.cc:348] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.681811 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:27.681939 5105 raft_consensus.cc:374] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.682025 5105 raft_consensus.cc:729] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.682623 5159 ts_tablet_manager.cc:1888] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:27.682072 5668 consensus_queue.cc:237] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.684774 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd27023b3f47f281dfb5474ea3a4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.685338 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b28bb9ac773c4bc685cb8193e574460b (table default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]) successfully deleted
I20240503 02:12:27.685585 5159 log.cc:1196] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/acdefb33fe6e4a9893dd66f6252395d7
W20240503 02:12:27.685716 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a because the tablet was not found. No further retry: Not found: Tablet not found: 9b0d8eb5f0d84bcf9b5b9e0ea1e8ca5a
I20240503 02:12:27.686959 5159 ts_tablet_manager.cc:1909] T acdefb33fe6e4a9893dd66f6252395d7 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:27.688004 5105 consensus_queue.cc:260] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:27.641029 4608 leader_election.cc:341] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:27.690095 5105 ts_tablet_manager.cc:1398] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.043s user 0.001s sys 0.002s
I20240503 02:12:27.690315 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 17a24b5c83b649e79644b08253d851d6 (DEFAULT_TABLE table=default.Table_7fe2fd4286f4411aad562211e4bd77ae [id=2073875662ee44ca99e5837ba4ace7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.692091 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:27.694640 4636 tablet_service.cc:1467] Processing CreateTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 (DEFAULT_TABLE table=default.Table_45f329757bf64154809c2a5b327f549f [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.690958 4670 raft_consensus.cc:2454] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:27.705358 5105 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:27.708891 5105 tablet_bootstrap.cc:654] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:27.710187 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 consensus state has no leader
I20240503 02:12:27.711592 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet aaa77e6c9a494fa2bdc74c5d725624cb
I20240503 02:12:27.715821 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a8465a99526148a0b0b88f92c5d1e3a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.717358 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8465a99526148a0b0b88f92c5d1e3a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.720988 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_19cd1048ca9d40cb9bfd8f1267a99e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.721083 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e73c3b3756c349dba71457b7d6a40ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.721153 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_25ebb22ce7fd40f7bd9828551cc26b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.721663 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b9f90c9e6ae3489b9809bed574c04644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.667150 5652 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
I20240503 02:12:27.661125 5613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:27.722633 5613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:12:27.722899 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19cd1048ca9d40cb9bfd8f1267a99e0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.705559 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 390) took 2632 ms. Trace:
I20240503 02:12:27.713342 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.724030 5159 rpcz_store.cc:276] 0503 02:12:25.061444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.061494 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:27.435423 (+2373929us) ts_tablet_manager.cc:1120] Deleting tablet acdefb33fe6e4a9893dd66f6252395d7
0503 02:12:27.435435 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.462660 (+ 27225us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.682421 (+219761us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.693580 (+ 11159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373731}
I20240503 02:12:27.725047 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 391) took 2654 ms. Trace:
I20240503 02:12:27.725244 5159 rpcz_store.cc:276] 0503 02:12:25.070499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.070556 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:27.724739 (+2654183us) ts_tablet_manager.cc:1120] Deleting tablet 8a90ebcf29004d1a925b69fa20998dd3
0503 02:12:27.724752 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.724985 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653919}
I20240503 02:12:27.726085 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 392) took 2420 ms. Trace:
I20240503 02:12:27.726406 5159 rpcz_store.cc:276] 0503 02:12:25.305489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.305544 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:27.725831 (+2420287us) ts_tablet_manager.cc:1120] Deleting tablet 43119471f91c4a1ba900f267c88db9a6
0503 02:12:27.725867 (+ 36us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.726028 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420058}
W20240503 02:12:27.726567 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e73c3b3756c349dba71457b7d6a40ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.727499 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 393) took 2390 ms. Trace:
I20240503 02:12:27.728056 5159 rpcz_store.cc:276] 0503 02:12:25.336964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.337022 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:27.727105 (+2390083us) ts_tablet_manager.cc:1120] Deleting tablet 5dbbfa2d1a49400c809d5b476036cdcd
0503 02:12:27.727139 (+ 34us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.727355 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389844}
I20240503 02:12:27.661619 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df63329dc2b149d9b08cea341b540d1d. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:27.730162 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:27.730530 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 397) took 2361 ms. Trace:
W20240503 02:12:27.733875 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9f90c9e6ae3489b9809bed574c04644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.724540 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet acdefb33fe6e4a9893dd66f6252395d7 (table default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]) successfully deleted
W20240503 02:12:27.734403 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25ebb22ce7fd40f7bd9828551cc26b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:27.734740 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3b72c20a4b5a4d2ea266f031733d1aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b72c20a4b5a4d2ea266f031733d1aab already in progress: creating tablet
W20240503 02:12:27.735003 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 23ccd25074db40fb96053b915125a425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ccd25074db40fb96053b915125a425 already in progress: creating tablet
W20240503 02:12:27.735275 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d39f042afb954daa923502ef12377bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d39f042afb954daa923502ef12377bd5 already in progress: creating tablet
W20240503 02:12:27.735529 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8a90ebcf29004d1a925b69fa20998dd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a90ebcf29004d1a925b69fa20998dd3 already in progress: creating tablet
W20240503 02:12:27.735771 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 43119471f91c4a1ba900f267c88db9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43119471f91c4a1ba900f267c88db9a6 already in progress: creating tablet
W20240503 02:12:27.735999 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5dbbfa2d1a49400c809d5b476036cdcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dbbfa2d1a49400c809d5b476036cdcd already in progress: creating tablet
W20240503 02:12:27.736389 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8241a88f5c3a47578c92343f9fb5f7cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8241a88f5c3a47578c92343f9fb5f7cd already in progress: creating tablet
I20240503 02:12:27.732228 5159 rpcz_store.cc:276] 0503 02:12:25.368966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.369050 (+ 84us) service_pool.cc:225] Handling call
0503 02:12:27.729704 (+2360654us) ts_tablet_manager.cc:1120] Deleting tablet 84b6e303e34a465caffed6c9e1e6bf74
0503 02:12:27.729717 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.730391 (+ 674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2360027}
W20240503 02:12:27.737512 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 84b6e303e34a465caffed6c9e1e6bf74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b6e303e34a465caffed6c9e1e6bf74 already in progress: creating tablet
I20240503 02:12:27.738268 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 396) took 2374 ms. Trace:
I20240503 02:12:27.743969 5159 rpcz_store.cc:276] 0503 02:12:25.363544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.369548 (+ 6004us) service_pool.cc:225] Handling call
0503 02:12:27.738087 (+2368539us) ts_tablet_manager.cc:1120] Deleting tablet 76b01907833d44f6a000a4de3a2a6e96
0503 02:12:27.738100 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.738229 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368280}
W20240503 02:12:27.745414 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 76b01907833d44f6a000a4de3a2a6e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76b01907833d44f6a000a4de3a2a6e96 already in progress: creating tablet
I20240503 02:12:27.746165 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 399) took 2362 ms. Trace:
I20240503 02:12:27.747180 5159 rpcz_store.cc:276] 0503 02:12:25.383596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.385623 (+ 2027us) service_pool.cc:225] Handling call
0503 02:12:27.745955 (+2360332us) ts_tablet_manager.cc:1120] Deleting tablet 8d0494ee4bd746069bfaf39bfb46687d
0503 02:12:27.745968 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.746128 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359697}
W20240503 02:12:27.748425 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8d0494ee4bd746069bfaf39bfb46687d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d0494ee4bd746069bfaf39bfb46687d already in progress: creating tablet
I20240503 02:12:27.749384 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 400) took 2353 ms. Trace:
I20240503 02:12:27.750186 5159 rpcz_store.cc:276] 0503 02:12:25.396210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.396319 (+ 109us) service_pool.cc:225] Handling call
0503 02:12:27.749210 (+2352891us) ts_tablet_manager.cc:1120] Deleting tablet a88710ac9062472f80bf6fcfb82793a0
0503 02:12:27.749225 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.749347 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343524}
W20240503 02:12:27.751494 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a88710ac9062472f80bf6fcfb82793a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88710ac9062472f80bf6fcfb82793a0 already in progress: creating tablet
I20240503 02:12:27.756129 5159 tablet_replica.cc:330] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:27.757302 5159 raft_consensus.cc:2229] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:27.758788 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.759374 5159 pending_rounds.cc:62] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
W20240503 02:12:27.759908 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.759866 5159 pending_rounds.cc:68] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420809047531520 op_type: NO_OP noop_request { }
I20240503 02:12:27.760025 5159 raft_consensus.cc:2866] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:27.760814 5159 raft_consensus.cc:2258] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:27.762542 5159 ts_tablet_manager.cc:1875] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:27.767612 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_923e68dde96e47489191170eb93b19ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.768420 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_923e68dde96e47489191170eb93b19ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.779826 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5506130048488caa3ee18ff7229835 (DEFAULT_TABLE table=default.Table_6bb9a205ce214c72a18394200f496a52 [id=f1d87a24dcb54d32bd591a31eec4f0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.782686 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 197bca686b5e4d6cb3e6c846ed8eb584 (DEFAULT_TABLE table=default.Table_c269ebd3c64a4b61964defe4593937ad [id=5ad9c30af1ee4df1b1dacbbc534d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.785308 4780 tablet_service.cc:1467] Processing CreateTablet for tablet ad17e65b0fc34236a19c7ebd077b75de (DEFAULT_TABLE table=default.Table_82701b51f62944de9442411ee0b446fd [id=fb09a538620346278dc8e0fd6c3ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.785310 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_73a988b03f62487b86ace12f89bcd124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.786093 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73a988b03f62487b86ace12f89bcd124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.787544 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_45f329757bf64154809c2a5b327f549f" } new_table_name: "default.Table_d286219eeee948669a9681bd7e501192" modify_external_catalogs: true
I20240503 02:12:27.788726 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 (DEFAULT_TABLE table=default.Table_301d83f516e84342a226f385300cef79 [id=8d41860986c04f2b9b365c0af0744299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.789594 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0af50eb2b1c949a0a6a71e64a894e8a5
I20240503 02:12:27.790097 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3
I20240503 02:12:27.790562 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 deleted and replaced by tablet b72bff3c50eb4ac6b1b2feb7250ba43e
I20240503 02:12:27.791353 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.791712 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 with delete_type TABLET_DATA_DELETED (replaced by tablet b72bff3c50eb4ac6b1b2feb7250ba43e) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.791929 4641 tablet_service.cc:1467] Processing CreateTablet for tablet 6624e145aac2412d98acf847e9e9c8b8 (DEFAULT_TABLE table=default.Table_ae1fa81b7e334c958fc7d781ecd7bc11 [id=340f7dfafe334dbdbed043aab0b2a97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.792495 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 (DEFAULT_TABLE table=default.Table_5920bfd139224d908c23f21d6a65308a [id=6c79ef76b01744f0a83455d15c08d34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.792865 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd (DEFAULT_TABLE table=default.Table_47b984d6c813457786ac8b9c4fad0255 [id=9ab5a5bf648e486a97d8ea223c90dc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.793314 4650 tablet_service.cc:1467] Processing CreateTablet for tablet a6de1b96c15c4b44809a03ef99e077ba (DEFAULT_TABLE table=default.Table_98865c1dc735482d9fdb50d75f612ac9 [id=f40c534a8761468f970216fb0260c33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.794163 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5506130048488caa3ee18ff7229835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.794824 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad17e65b0fc34236a19c7ebd077b75de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.795395 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197bca686b5e4d6cb3e6c846ed8eb584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.796175 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 367de4ff0a6f4abfa2924a070f387cbd
I20240503 02:12:27.796774 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 620204f7cb35442f87eae45f84beeee1
I20240503 02:12:27.797004 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1ce9d746acfe4e9aaea600293470d373
I20240503 02:12:27.797964 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38614de3642148aa82e5108d9160f537. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.806967 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce9d746acfe4e9aaea600293470d373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.807415 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.807699 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 620204f7cb35442f87eae45f84beeee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:27.816335 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_daa34adc9b654cc2a6e834aeb6e0a9ae" } new_table_name: "default.Table_80c858c6cd0b407da24b467f885e2b76" modify_external_catalogs: true
I20240503 02:12:27.816390 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e2ed0b0221564739b092ebd62980fc94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.816969 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2ed0b0221564739b092ebd62980fc94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.817220 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c269ebd3c64a4b61964defe4593937ad" } new_table_name: "default.Table_e37994d9ab31473f8022a677b6949bde" modify_external_catalogs: true
I20240503 02:12:27.818949 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f3b793f1555b46458c1b7bd8682deeea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.819617 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b793f1555b46458c1b7bd8682deeea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:27.821291 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c919bda1eaaa4e52828aac3ef337f3e5 consensus state has no leader
I20240503 02:12:27.823195 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 984fe4640980407aa03c16c14a1303c1 (DEFAULT_TABLE table=default.Table_6623c5960394449394b0bc3dae42bdd6 [id=4915cc5437ad47798f1763ece3572049]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.823817 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984fe4640980407aa03c16c14a1303c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.825115 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868a6ad6cc1a40f091b4516676c24c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.833107 4780 tablet_service.cc:1467] Processing CreateTablet for tablet c2141d4c11854a859f1ffdc5ccb953bd (DEFAULT_TABLE table=default.Table_c3b4aa722ff64856923050d70b17d823 [id=63857008baf244ce828b234755fe9503]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.834025 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2141d4c11854a859f1ffdc5ccb953bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.836266 5106 tablet_bootstrap.cc:654] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:27.837409 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 197bca686b5e4d6cb3e6c846ed8eb584 consensus state has no leader
W20240503 02:12:27.837483 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 81401.1us
I20240503 02:12:27.839006 4812 raft_consensus.cc:3037] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.842850 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6de1b96c15c4b44809a03ef99e077ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.851716 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c18cecefab4b6caed0039d09cc1890 with delete_type TABLET_DATA_DELETED (replaced by tablet 3aad7d6dcf2e4009872ce156c536f38e) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.853102 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af50eb2b1c949a0a6a71e64a894e8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.858886 5159 ts_tablet_manager.cc:1888] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:27.865365 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.865868 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624e145aac2412d98acf847e9e9c8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.871042 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5506130048488caa3ee18ff7229835 (DEFAULT_TABLE table=default.Table_6bb9a205ce214c72a18394200f496a52 [id=f1d87a24dcb54d32bd591a31eec4f0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.874483 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 197bca686b5e4d6cb3e6c846ed8eb584 (DEFAULT_TABLE table=default.Table_c269ebd3c64a4b61964defe4593937ad [id=5ad9c30af1ee4df1b1dacbbc534d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.867424 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 88913ce4890c4f528f47163b6c0f04fe (DEFAULT_TABLE table=default.Table_63c3234b89e94ea4a79400239439762d [id=4b98d85c043d4660919e91032e6f4069]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.878492 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88913ce4890c4f528f47163b6c0f04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.869946 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c (DEFAULT_TABLE table=default.Table_186402d8f9e2484c89977667d29d1ff9 [id=415ffe34b0784872930d63674d5a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.881296 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e08c10c15f41878202cef0c8d5ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.883679 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c919bda1eaaa4e52828aac3ef337f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.894903 5159 log.cc:1196] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:27.895793 5159 ts_tablet_manager.cc:1909] T b28bb9ac773c4bc685cb8193e574460b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:27.898908 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197bca686b5e4d6cb3e6c846ed8eb584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.877817 5105 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:27.899222 5105 ts_tablet_manager.cc:1367] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.205s user 0.005s sys 0.010s
I20240503 02:12:27.900329 5105 raft_consensus.cc:348] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.900648 5105 raft_consensus.cc:374] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.900765 5105 raft_consensus.cc:729] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.898541 4784 tablet_service.cc:1467] Processing CreateTablet for tablet fe20d0ca30f845618e0a20aa85dafbde (DEFAULT_TABLE table=default.Table_252b426637d647128fd41a25217a1cd3 [id=14c5cd8c02fe49bca90a36e1851e7288]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.901317 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20d0ca30f845618e0a20aa85dafbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.901893 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.903069 4642 tablet_service.cc:1467] Processing CreateTablet for tablet ad17e65b0fc34236a19c7ebd077b75de (DEFAULT_TABLE table=default.Table_82701b51f62944de9442411ee0b446fd [id=fb09a538620346278dc8e0fd6c3ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.905097 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 404) took 2430 ms. Trace:
I20240503 02:12:27.905320 5159 rpcz_store.cc:276] 0503 02:12:25.474541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.474595 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:27.756047 (+2281452us) ts_tablet_manager.cc:1120] Deleting tablet b28bb9ac773c4bc685cb8193e574460b
0503 02:12:27.756060 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.819507 (+ 63447us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.858704 (+ 39197us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.905052 (+ 46348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281215}
I20240503 02:12:27.906020 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b28bb9ac773c4bc685cb8193e574460b (table default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]) successfully deleted
I20240503 02:12:27.906628 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 411) took 2098 ms. Trace:
I20240503 02:12:27.906795 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5506130048488caa3ee18ff7229835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.906800 5159 rpcz_store.cc:276] 0503 02:12:25.808613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.808676 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:27.906458 (+2097782us) ts_tablet_manager.cc:1120] Deleting tablet ea5e4db5204e4acf8719100d02d533fb
0503 02:12:27.906469 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:27.906590 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2097445}
I20240503 02:12:27.915724 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_37e1baa613ca4b10bb2727227e93163b" } new_table_name: "default.Table_fb11bd168d9b4a22940b384400a5c6d5" modify_external_catalogs: true
I20240503 02:12:27.917613 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_252b426637d647128fd41a25217a1cd3" } new_table_name: "default.Table_a145e05d0eed4f8db0706ba2b2f6025e" modify_external_catalogs: true
I20240503 02:12:27.919198 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_47b984d6c813457786ac8b9c4fad0255" } modify_external_catalogs: true
I20240503 02:12:27.919423 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_47b984d6c813457786ac8b9c4fad0255" } modify_external_catalogs: true
I20240503 02:12:27.921188 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad17e65b0fc34236a19c7ebd077b75de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.913009 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1f8b305ac6424a8bc6a9cf25eec587 (DEFAULT_TABLE table=default.Table_a3621f453c4d4329acee2b8d139c662a [id=8b91af0af2864a4ebcc056e641af86b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.924351 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1f8b305ac6424a8bc6a9cf25eec587. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:27.924470 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48e713ee37d042309166b0f118e14986 consensus state has no leader
I20240503 02:12:27.924769 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.928658 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5a915d760ba14a31895a5dc7ec5b34fd
I20240503 02:12:27.914958 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 584694f1cc284a718f0e5fa15785d649 (DEFAULT_TABLE table=default.Table_ea8c1983dad340faaa278bfe0d6ea103 [id=b771a5d963eb4ac2b0065061b9feadac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.929283 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584694f1cc284a718f0e5fa15785d649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.929967 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:27.930328 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe20d0ca30f845618e0a20aa85dafbde consensus state has no leader
I20240503 02:12:27.932713 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a915d760ba14a31895a5dc7ec5b34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.917711 5105 consensus_queue.cc:260] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.933534 5105 ts_tablet_manager.cc:1398] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.034s user 0.000s sys 0.002s
W20240503 02:12:27.934087 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ea5e4db5204e4acf8719100d02d533fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5e4db5204e4acf8719100d02d533fb already in progress: creating tablet
I20240503 02:12:27.937287 5611 raft_consensus.cc:482] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.937539 5611 raft_consensus.cc:504] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:27.938478 5611 leader_election.cc:288] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:27.939750 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 (DEFAULT_TABLE table=default.Table_88f4791ae6054ec5becf0d13aec8be47 [id=0cae1c89bb5048af85cdb5d6ef92489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.943221 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 (DEFAULT_TABLE table=default.Table_997fff88119a49a5b51f25324fe4f546 [id=7a6371c00571476eba9ef053d8cba825]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.955083 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 8135907b78284d7ca74396cc77dd0612 (DEFAULT_TABLE table=default.Table_3e1aa615a80d47258acd2c493d6cb299 [id=ce9c0a316ba0498781109f1fba39bbd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.957454 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4aad131b884c54b69b3d1d9420662c (DEFAULT_TABLE table=default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8 [id=c3eeec6ee1b74545af2b4f26849883f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.957823 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a24b5c83b649e79644b08253d851d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.958746 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 6b15997a94b74ad3808d965acfb5d155 (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.959803 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a2518b080992412aa7ae7abdd643bdf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.960469 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2518b080992412aa7ae7abdd643bdf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.961876 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bcefea9ec04543dc8b079eb3822d739f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.962416 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcefea9ec04543dc8b079eb3822d739f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.967761 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:27.968076 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5c4057da7da24b3e8c9e3e1a4875666e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:27.968206 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6a0d5b09dfb04a1892fb248b676a923b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.968463 4746 leader_election.cc:341] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:27.968679 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c4057da7da24b3e8c9e3e1a4875666e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:27.968746 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a0d5b09dfb04a1892fb248b676a923b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.969235 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e0f6872c37c746fab51a59df01834e38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.969704 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0f6872c37c746fab51a59df01834e38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.973841 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 with delete_type TABLET_DATA_DELETED (replaced by tablet b72bff3c50eb4ac6b1b2feb7250ba43e) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:27.993471 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0e5ff92986ff447dbc2c9e69a628e8fe" } modify_external_catalogs: true
I20240503 02:12:27.993813 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0e5ff92986ff447dbc2c9e69a628e8fe" } modify_external_catalogs: true
I20240503 02:12:27.995258 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_081e142b6f9e45b5b4cb3987b8f31ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:27.995877 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_081e142b6f9e45b5b4cb3987b8f31ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:27.997324 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 43e8fc6409b5448eaa567879c5b7d504 (DEFAULT_TABLE table=default.Table_6d6c3b05fef34441af532f58a3fb5f6a [id=6d141cfe014740069cf37d1d246f2efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.014192 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:28.027542 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce9d746acfe4e9aaea600293470d373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:28.028072 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 984fe4640980407aa03c16c14a1303c1 (DEFAULT_TABLE table=default.Table_6623c5960394449394b0bc3dae42bdd6 [id=4915cc5437ad47798f1763ece3572049]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.028676 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984fe4640980407aa03c16c14a1303c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.072022 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 620204f7cb35442f87eae45f84beeee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:28.076841 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290a61c21fa456eb3c4f9b0dbebc864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.078052 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 410) took 2270 ms. Trace:
I20240503 02:12:28.078188 5159 rpcz_store.cc:276] 0503 02:12:25.808018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.809284 (+ 1266us) service_pool.cc:225] Handling call
0503 02:12:27.934480 (+2125196us) ts_tablet_manager.cc:1120] Deleting tablet 79b8db679ba24ac6a93be591c0513ab0
0503 02:12:28.077836 (+143356us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.078001 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124900}
I20240503 02:12:28.078574 5159 tablet_replica.cc:330] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:28.078816 5159 raft_consensus.cc:2229] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:28.079018 5159 raft_consensus.cc:2258] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:28.079702 5159 ts_tablet_manager.cc:1875] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:28.082576 4319 kernel_stack_watchdog.cc:198] Thread 4331 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 314ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ff4ac000980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7ff4a8c6ed11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7ff4a8c8f7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7ff4a8c8b3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7ff4a8c904ff kudu::ThreadPool::Submit() at ??:0
@ 0x7ff4af8ca682 kudu::rpc::ReactorThread::StartConnectionNegotiation() at ??:0
@ 0x7ff4af8c9b8b kudu::rpc::ReactorThread::RegisterConnection() at ??:0
@ 0x7ff4af8db00e kudu::rpc::RegisterConnectionTask::Run() at ??:0
@ 0x7ff4af8c969d kudu::rpc::ReactorThread::AsyncHandler() at ??:0
@ 0x7ff4af8dbf45 ev::base<>::method_thunk<>() at ??:0
@ 0x7ff4a747d699 ev_invoke_pending at ??:0
@ 0x7ff4af8c6ba5 kudu::rpc::ReactorThread::InvokePendingCb() at ??:0
@ 0x7ff4a747e5a1 ev_run at ??:0
I20240503 02:12:28.083196 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c61bf67c8a47fb9ef23bcd6c58c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.083464 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e8fc6409b5448eaa567879c5b7d504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.090229 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4aad131b884c54b69b3d1d9420662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.098407 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8135907b78284d7ca74396cc77dd0612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.098807 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b15997a94b74ad3808d965acfb5d155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.130004 5688 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 02:12:28.143049 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet acdefb33fe6e4a9893dd66f6252395d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:19 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.143460 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.146152 5159 ts_tablet_manager.cc:1888] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:28.146325 5159 log.cc:1196] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/46560ac6d94943d2a8fb3b3904f2b5b6
I20240503 02:12:28.146801 5159 ts_tablet_manager.cc:1909] T 46560ac6d94943d2a8fb3b3904f2b5b6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:28.147027 4640 tablet_service.cc:1467] Processing CreateTablet for tablet c2141d4c11854a859f1ffdc5ccb953bd (DEFAULT_TABLE table=default.Table_c3b4aa722ff64856923050d70b17d823 [id=63857008baf244ce828b234755fe9503]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.148101 5105 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:28.149868 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.150807 5105 tablet_bootstrap.cc:654] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.152247 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 412) took 2342 ms. Trace:
I20240503 02:12:28.152560 5159 rpcz_store.cc:276] 0503 02:12:25.809649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.809709 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:28.078501 (+2268792us) ts_tablet_manager.cc:1120] Deleting tablet 46560ac6d94943d2a8fb3b3904f2b5b6
0503 02:12:28.078514 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.122368 (+ 43854us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.146004 (+ 23636us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.152186 (+ 6182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2268581}
I20240503 02:12:28.153352 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 409) took 2346 ms. Trace:
I20240503 02:12:28.153441 5159 rpcz_store.cc:276] 0503 02:12:25.807014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:25.810602 (+ 3588us) service_pool.cc:225] Handling call
0503 02:12:28.153100 (+2342498us) ts_tablet_manager.cc:1120] Deleting tablet 94559cfc32704eae96b121f165686edc
0503 02:12:28.153113 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.153268 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342259}
I20240503 02:12:28.154173 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 414) took 1960 ms. Trace:
I20240503 02:12:28.154275 5159 rpcz_store.cc:276] 0503 02:12:26.194247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.194327 (+ 80us) service_pool.cc:225] Handling call
0503 02:12:28.153993 (+1959666us) ts_tablet_manager.cc:1120] Deleting tablet 3db94fb9d53044d796e2973143d888fb
0503 02:12:28.154006 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.154112 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952971}
W20240503 02:12:28.154512 5103 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.142s sys 0.211s
I20240503 02:12:28.158054 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 413) took 2040 ms. Trace:
I20240503 02:12:28.158155 5159 rpcz_store.cc:276] 0503 02:12:26.117331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.117431 (+ 100us) service_pool.cc:225] Handling call
0503 02:12:28.157892 (+2040461us) ts_tablet_manager.cc:1120] Deleting tablet a0e1c2fa9cb3481dafb9d7f1e826a30c
0503 02:12:28.157902 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.158004 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1954181}
I20240503 02:12:28.159677 5105 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:28.159881 5105 ts_tablet_manager.cc:1367] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.002s
I20240503 02:12:28.160892 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5506130048488caa3ee18ff7229835 (DEFAULT_TABLE table=default.Table_6bb9a205ce214c72a18394200f496a52 [id=f1d87a24dcb54d32bd591a31eec4f0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.161170 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 431) took 1885 ms. Trace:
I20240503 02:12:28.161252 5159 rpcz_store.cc:276] 0503 02:12:26.276122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.276177 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:28.161013 (+1884836us) ts_tablet_manager.cc:1120] Deleting tablet 3aad7d6dcf2e4009872ce156c536f38e
0503 02:12:28.161022 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.161122 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884281}
I20240503 02:12:28.162441 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 436) took 1879 ms. Trace:
I20240503 02:12:28.162537 5159 rpcz_store.cc:276] 0503 02:12:26.283001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.283227 (+ 226us) service_pool.cc:225] Handling call
0503 02:12:28.162294 (+1879067us) ts_tablet_manager.cc:1120] Deleting tablet 043ba14eaa5b46319b0b88457d2a5f45
0503 02:12:28.162303 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.162389 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878550}
I20240503 02:12:28.163519 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 427) took 1893 ms. Trace:
I20240503 02:12:28.163615 5159 rpcz_store.cc:276] 0503 02:12:26.270077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.270145 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:28.163360 (+1893215us) ts_tablet_manager.cc:1120] Deleting tablet 905ca3be01e344d38350a58ac9f0d321
0503 02:12:28.163372 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.163474 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877344}
I20240503 02:12:28.174782 5611 raft_consensus.cc:482] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.175019 5611 raft_consensus.cc:504] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.175971 5611 leader_election.cc:288] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:28.160780 5105 raft_consensus.cc:348] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.177934 5105 raft_consensus.cc:374] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.178050 5105 raft_consensus.cc:729] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.178365 5105 consensus_queue.cc:260] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.179059 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 430) took 1905 ms. Trace:
I20240503 02:12:28.179162 5159 rpcz_store.cc:276] 0503 02:12:26.274107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.274163 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:28.178838 (+1904675us) ts_tablet_manager.cc:1120] Deleting tablet cd66f88a24b74f2c9b2050af081d4601
0503 02:12:28.178851 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.179008 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886303}
I20240503 02:12:28.179658 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 428) took 1909 ms. Trace:
I20240503 02:12:28.179744 5159 rpcz_store.cc:276] 0503 02:12:26.270203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.271271 (+ 1068us) service_pool.cc:225] Handling call
0503 02:12:28.179438 (+1908167us) ts_tablet_manager.cc:1120] Deleting tablet 1436dd3df7284f9bb379236eb9d76836
0503 02:12:28.179449 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.179603 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862917}
I20240503 02:12:28.180120 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 442) took 1562 ms. Trace:
I20240503 02:12:28.180277 5159 rpcz_store.cc:276] 0503 02:12:26.617755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.618176 (+ 421us) service_pool.cc:225] Handling call
0503 02:12:28.179966 (+1561790us) ts_tablet_manager.cc:1120] Deleting tablet 2ecfd29eddde4c39b7661b1b7d96a08e
0503 02:12:28.179975 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.180068 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560936}
I20240503 02:12:28.180640 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 447) took 1534 ms. Trace:
I20240503 02:12:28.180720 5159 rpcz_store.cc:276] 0503 02:12:26.646041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.646114 (+ 73us) service_pool.cc:225] Handling call
0503 02:12:28.180505 (+1534391us) ts_tablet_manager.cc:1120] Deleting tablet 3f9994ca5aea48d3bc25c097dddba3c3
0503 02:12:28.180514 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.180589 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519044}
I20240503 02:12:28.181017 5159 tablet_replica.cc:330] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:28.181203 5159 raft_consensus.cc:2229] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:28.185096 5105 ts_tablet_manager.cc:1398] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240503 02:12:28.185487 5105 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:28.186586 5103 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.743s user 0.142s sys 0.211s
I20240503 02:12:28.186555 5666 raft_consensus.cc:482] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.186764 5666 raft_consensus.cc:504] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.187778 5666 leader_election.cc:288] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:28.188033 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.188294 4529 raft_consensus.cc:2454] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:28.189505 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.162523 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.194557 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 197bca686b5e4d6cb3e6c846ed8eb584 (DEFAULT_TABLE table=default.Table_c269ebd3c64a4b61964defe4593937ad [id=5ad9c30af1ee4df1b1dacbbc534d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.195219 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 with delete_type TABLET_DATA_DELETED (replaced by tablet b72bff3c50eb4ac6b1b2feb7250ba43e) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.195947 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.196110 5103 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:28.196641 5114 tablet_replica.cc:330] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:28.196796 4746 leader_election.cc:302] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:28.197562 5611 raft_consensus.cc:2781] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.197698 5611 raft_consensus.cc:482] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.197733 5159 raft_consensus.cc:2258] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:28.197810 5611 raft_consensus.cc:3037] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:28.200265 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 79b8db679ba24ac6a93be591c0513ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79b8db679ba24ac6a93be591c0513ab0 already in progress: creating tablet
I20240503 02:12:28.201535 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 46560ac6d94943d2a8fb3b3904f2b5b6 (table default.Table_23083e3bb77444ec82de47917ee95f01 [id=b4c01b578b424aa2871f82e607dba76d]) successfully deleted
W20240503 02:12:28.201824 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 94559cfc32704eae96b121f165686edc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94559cfc32704eae96b121f165686edc already in progress: creating tablet
I20240503 02:12:28.203303 5103 ts_tablet_manager.cc:1367] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.796s user 0.143s sys 0.212s
W20240503 02:12:28.203336 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3db94fb9d53044d796e2973143d888fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db94fb9d53044d796e2973143d888fb already in progress: creating tablet
W20240503 02:12:28.203603 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e1c2fa9cb3481dafb9d7f1e826a30c already in progress: creating tablet
I20240503 02:12:28.203110 5611 raft_consensus.cc:504] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.206390 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3aad7d6dcf2e4009872ce156c536f38e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aad7d6dcf2e4009872ce156c536f38e already in progress: creating tablet
I20240503 02:12:28.206388 5611 leader_election.cc:288] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:28.206697 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 043ba14eaa5b46319b0b88457d2a5f45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 043ba14eaa5b46319b0b88457d2a5f45 already in progress: creating tablet
I20240503 02:12:28.203740 4508 tablet_service.cc:1467] Processing CreateTablet for tablet ad17e65b0fc34236a19c7ebd077b75de (DEFAULT_TABLE table=default.Table_82701b51f62944de9442411ee0b446fd [id=fb09a538620346278dc8e0fd6c3ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.208956 5159 ts_tablet_manager.cc:1875] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:28.210789 5114 raft_consensus.cc:2229] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:28.210989 5105 tablet_bootstrap.cc:654] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:28.211560 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 905ca3be01e344d38350a58ac9f0d321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905ca3be01e344d38350a58ac9f0d321 already in progress: creating tablet
W20240503 02:12:28.211884 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet cd66f88a24b74f2c9b2050af081d4601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd66f88a24b74f2c9b2050af081d4601 already in progress: creating tablet
I20240503 02:12:28.214413 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9327a9e5472b4eda854320c3a95f8e56" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.215142 5103 raft_consensus.cc:348] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.200865 5401 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.011s sys 0.000s
I20240503 02:12:28.220348 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0cc3309db13041969f6cbdf461a53f8f" } modify_external_catalogs: true
I20240503 02:12:28.220727 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_77eae5a54c4c440a9928ae87553755be" } modify_external_catalogs: true
I20240503 02:12:28.221661 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0cc3309db13041969f6cbdf461a53f8f" } modify_external_catalogs: true
I20240503 02:12:28.221753 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 607e49cde2fd4aab9f44921d3c3921a5
I20240503 02:12:28.221633 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ac2cdb3ebec4a2795b7fa6e47dc56b8" } modify_external_catalogs: true
I20240503 02:12:28.222369 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ac2cdb3ebec4a2795b7fa6e47dc56b8" } modify_external_catalogs: true
I20240503 02:12:28.221607 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_77eae5a54c4c440a9928ae87553755be" } modify_external_catalogs: true
I20240503 02:12:28.227103 5611 raft_consensus.cc:482] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.227519 5611 raft_consensus.cc:504] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.228588 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1436dd3df7284f9bb379236eb9d76836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1436dd3df7284f9bb379236eb9d76836 already in progress: creating tablet
I20240503 02:12:28.229362 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_466756ec40864212abe6bd1a32875b14" } new_table_name: "default.Table_d693ea577510429c96e04408da3828e1" modify_external_catalogs: true
I20240503 02:12:28.229735 5611 leader_election.cc:288] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:28.229748 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_01fcdbe2fef14f7284c1f68c4f72e92c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:28.231199 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_aeb4bc27f5dc47fca47f2316393e35c2" } new_table_name: "default.Table_5956db05373844bcafa9952fa9f6e513" modify_external_catalogs: true
I20240503 02:12:28.233841 4503 tablet_service.cc:1467] Processing CreateTablet for tablet c2141d4c11854a859f1ffdc5ccb953bd (DEFAULT_TABLE table=default.Table_c3b4aa722ff64856923050d70b17d823 [id=63857008baf244ce828b234755fe9503]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.234198 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2ecfd29eddde4c39b7661b1b7d96a08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ecfd29eddde4c39b7661b1b7d96a08e already in progress: creating tablet
I20240503 02:12:28.235339 5114 raft_consensus.cc:2258] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:28.233711 4529 raft_consensus.cc:2454] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
W20240503 02:12:28.236515 5589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.020s sys 0.030s
I20240503 02:12:28.237035 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 984fe4640980407aa03c16c14a1303c1 (DEFAULT_TABLE table=default.Table_6623c5960394449394b0bc3dae42bdd6 [id=4915cc5437ad47798f1763ece3572049]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.237413 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce9d746acfe4e9aaea600293470d373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.234357 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6de1b96c15c4b44809a03ef99e077ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.237480 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b3dcd45817264953aa80592e5bd4c4e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:28.238904 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ff6cfc8e7a2d4b8a8f691caa77265f87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:28.239394 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6cfc8e7a2d4b8a8f691caa77265f87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:28.239519 5105 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:28.239713 5105 ts_tablet_manager.cc:1367] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.002s
I20240503 02:12:28.240625 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e57bbe28945b49eb9a52ef6f4170ca13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:28.241117 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e57bbe28945b49eb9a52ef6f4170ca13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:28.242441 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3f9994ca5aea48d3bc25c097dddba3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f9994ca5aea48d3bc25c097dddba3c3 already in progress: creating tablet
I20240503 02:12:28.242507 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 88913ce4890c4f528f47163b6c0f04fe (DEFAULT_TABLE table=default.Table_63c3234b89e94ea4a79400239439762d [id=4b98d85c043d4660919e91032e6f4069]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.244246 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3dcd45817264953aa80592e5bd4c4e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:28.234010 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f0eecd86b1b2414f927cd22a6195fff2" } new_table_name: "default.Table_54ab74ddb0d94515895951df323ebd48" modify_external_catalogs: true
I20240503 02:12:28.253443 5105 raft_consensus.cc:348] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.253777 5105 raft_consensus.cc:374] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.253890 5105 raft_consensus.cc:729] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:28.255122 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01fcdbe2fef14f7284c1f68c4f72e92c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:28.257838 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2141d4c11854a859f1ffdc5ccb953bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.265301 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.259369 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9327a9e5472b4eda854320c3a95f8e56" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.268244 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 584694f1cc284a718f0e5fa15785d649 (DEFAULT_TABLE table=default.Table_ea8c1983dad340faaa278bfe0d6ea103 [id=b771a5d963eb4ac2b0065061b9feadac]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.268741 5682 log.cc:924] Time spent T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20240503 02:12:28.269997 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88913ce4890c4f528f47163b6c0f04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.271241 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584694f1cc284a718f0e5fa15785d649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.655833 5432 rpcz_store.cc:276] 0503 02:12:24.609948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:24.609998 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:26.520023 (+1910025us) raft_consensus.cc:1457] Updating replica for 363 ops
0503 02:12:26.540316 (+ 20293us) raft_consensus.cc:1520] Early marking committed up to index 115
0503 02:12:26.559872 (+ 19556us) raft_consensus.cc:1525] Triggering prepare for 363 ops
0503 02:12:26.560377 (+ 505us) write_op.cc:270] Start()
0503 02:12:26.560431 (+ 54us) write_op.cc:276] Timestamp: P: 1714702338626969 usec, L: 0
0503 02:12:26.567237 (+ 6806us) write_op.cc:270] Start()
0503 02:12:26.567290 (+ 53us) write_op.cc:276] Timestamp: P: 1714702338627920 usec, L: 0
0503 02:12:26.568418 (+ 1128us) write_op.cc:270] Start()
0503 02:12:26.568467 (+ 49us) write_op.cc:276] Timestamp: P: 1714702338634563 usec, L: 0
0503 02:12:26.569287 (+ 820us) write_op.cc:270] Start()
0503 02:12:26.569319 (+ 32us) write_op.cc:276] Timestamp: P: 1714702338654428 usec, L: 0
0503 02:12:26.569996 (+ 677us) write_op.cc:270] Start()
0503 02:12:26.570027 (+ 31us) write_op.cc:276] Timestamp: P: 1714702338655670 usec, L: 0
0503 02:12:26.570680 (+ 653us) write_op.cc:270] Start()
0503 02:12:26.570710 (+ 30us) write_op.cc:276] Timestamp: P: 1714702338656714 usec, L: 0
0503 02:12:26.581929 (+ 11219us) write_op.cc:270] Start()
0503 02:12:26.582019 (+ 90us) write_op.cc:276] Timestamp: P: 1714702338658366 usec, L: 0
0503 02:12:26.582784 (+ 765us) write_op.cc:270] Start()
0503 02:12:26.582819 (+ 35us) write_op.cc:276] Timestamp: P: 1714702338659529 usec, L: 0
0503 02:12:26.583784 (+ 965us) write_op.cc:270] Start()
0503 02:12:26.583819 (+ 35us) write_op.cc:276] Timestamp: P: 1714702338660464 usec, L: 0
0503 02:12:26.584642 (+ 823us) write_op.cc:270] Start()
0503 02:12:26.584676 (+ 34us) write_op.cc:276] Timestamp: P: 1714702338661407 usec, L: 0
0503 02:12:26.585416 (+ 740us) write_op.cc:270] Start()
0503 02:12:26.585446 (+ 30us) write_op.cc:276] Timestamp: P: 1714702338662342 usec, L: 0
0503 02:12:26.586187 (+ 741us) write_op.cc:270] Start()
0503 02:12:26.586235 (+ 48us) write_op.cc:276] Timestamp: P: 1714702338663177 usec, L: 0
0503 02:12:26.587120 (+ 885us) write_op.cc:270] Start()
0503 02:12:26.587154 (+ 34us) write_op.cc:276] Timestamp: P: 1714702338692534 usec, L: 0
0503 02:12:26.588085 (+ 931us) write_op.cc:270] Start()
0503 02:12:26.588120 (+ 35us) write_op.cc:276] Timestamp: P: 1714702338708305 usec, L: 0
0503 02:12:26.591480 (+ 3360us) write_op.cc:270] Start()
0503 02:12:26.591520 (+ 40us) write_op.cc:276] Timestamp: P: 1714702338718169 usec, L: 0
0503 02:12:26.591714 (+ 194us) write_op.cc:270] Start()
0503 02:12:26.591739 (+ 25us) write_op.cc:276] Timestamp: P: 1714702338722876 usec, L: 0
0503 02:12:26.591912 (+ 173us) write_op.cc:270] Start()
0503 02:12:26.591950 (+ 38us) write_op.cc:276] Timestamp: P: 1714702338735238 usec, L: 0
0503 02:12:26.592115 (+ 165us) write_op.cc:270] Start()
0503 02:12:26.592139 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338752588 usec, L: 0
0503 02:12:26.592364 (+ 225us) write_op.cc:270] Start()
0503 02:12:26.592398 (+ 34us) write_op.cc:276] Timestamp: P: 1714702338757845 usec, L: 0
0503 02:12:26.592542 (+ 144us) write_op.cc:270] Start()
0503 02:12:26.592566 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338787531 usec, L: 0
0503 02:12:26.592718 (+ 152us) write_op.cc:270] Start()
0503 02:12:26.592741 (+ 23us) write_op.cc:276] Timestamp: P: 1714702338809630 usec, L: 0
0503 02:12:26.592897 (+ 156us) write_op.cc:270] Start()
0503 02:12:26.592943 (+ 46us) write_op.cc:276] Timestamp: P: 1714702338810576 usec, L: 0
0503 02:12:26.593094 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.593118 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338838044 usec, L: 0
0503 02:12:26.593325 (+ 207us) write_op.cc:270] Start()
0503 02:12:26.593351 (+ 26us) write_op.cc:276] Timestamp: P: 1714702338853320 usec, L: 0
0503 02:12:26.595761 (+ 2410us) write_op.cc:270] Start()
0503 02:12:26.595813 (+ 52us) write_op.cc:276] Timestamp: P: 1714702338854992 usec, L: 0
0503 02:12:26.596034 (+ 221us) write_op.cc:270] Start()
0503 02:12:26.596061 (+ 27us) write_op.cc:276] Timestamp: P: 1714702338859001 usec, L: 0
0503 02:12:26.596259 (+ 198us) write_op.cc:270] Start()
0503 02:12:26.596284 (+ 25us) write_op.cc:276] Timestamp: P: 1714702338864515 usec, L: 0
0503 02:12:26.596511 (+ 227us) write_op.cc:270] Start()
0503 02:12:26.596539 (+ 28us) write_op.cc:276] Timestamp: P: 1714702338875094 usec, L: 0
0503 02:12:26.596707 (+ 168us) write_op.cc:270] Start()
0503 02:12:26.596732 (+ 25us) write_op.cc:276] Timestamp: P: 1714702338878646 usec, L: 0
0503 02:12:26.596894 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.596937 (+ 43us) write_op.cc:276] Timestamp: P: 1714702338879324 usec, L: 0
0503 02:12:26.597093 (+ 156us) write_op.cc:270] Start()
0503 02:12:26.597117 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338899436 usec, L: 0
0503 02:12:26.597302 (+ 185us) write_op.cc:270] Start()
0503 02:12:26.601148 (+ 3846us) write_op.cc:276] Timestamp: P: 1714702338900104 usec, L: 0
0503 02:12:26.601433 (+ 285us) write_op.cc:270] Start()
0503 02:12:26.601460 (+ 27us) write_op.cc:276] Timestamp: P: 1714702338903121 usec, L: 0
0503 02:12:26.601615 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.601638 (+ 23us) write_op.cc:276] Timestamp: P: 1714702338903728 usec, L: 0
0503 02:12:26.601794 (+ 156us) write_op.cc:270] Start()
0503 02:12:26.601818 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338904293 usec, L: 0
0503 02:12:26.601972 (+ 154us) write_op.cc:270] Start()
0503 02:12:26.601995 (+ 23us) write_op.cc:276] Timestamp: P: 1714702338920299 usec, L: 0
0503 02:12:26.602361 (+ 366us) write_op.cc:270] Start()
0503 02:12:26.602415 (+ 54us) write_op.cc:276] Timestamp: P: 1714702338939890 usec, L: 0
0503 02:12:26.602614 (+ 199us) write_op.cc:270] Start()
0503 02:12:26.602640 (+ 26us) write_op.cc:276] Timestamp: P: 1714702338944473 usec, L: 0
0503 02:12:26.602819 (+ 179us) write_op.cc:270] Start()
0503 02:12:26.602844 (+ 25us) write_op.cc:276] Timestamp: P: 1714702338945712 usec, L: 0
0503 02:12:26.603003 (+ 159us) write_op.cc:270] Start()
0503 02:12:26.603034 (+ 31us) write_op.cc:276] Timestamp: P: 1714702338947532 usec, L: 0
0503 02:12:26.603192 (+ 158us) write_op.cc:270] Start()
0503 02:12:26.603228 (+ 36us) write_op.cc:276] Timestamp: P: 1714702338968810 usec, L: 0
0503 02:12:26.603370 (+ 142us) write_op.cc:270] Start()
0503 02:12:26.603393 (+ 23us) write_op.cc:276] Timestamp: P: 1714702338972565 usec, L: 0
0503 02:12:26.603550 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.603574 (+ 24us) write_op.cc:276] Timestamp: P: 1714702338984808 usec, L: 0
0503 02:12:26.603714 (+ 140us) write_op.cc:270] Start()
0503 02:12:26.603737 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339013512 usec, L: 0
0503 02:12:26.603891 (+ 154us) write_op.cc:270] Start()
0503 02:12:26.603916 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339025942 usec, L: 0
0503 02:12:26.604062 (+ 146us) write_op.cc:270] Start()
0503 02:12:26.604086 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339026675 usec, L: 0
0503 02:12:26.604237 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.604261 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339039243 usec, L: 0
0503 02:12:26.604437 (+ 176us) write_op.cc:270] Start()
0503 02:12:26.604459 (+ 22us) write_op.cc:276] Timestamp: P: 1714702339048296 usec, L: 0
0503 02:12:26.604764 (+ 305us) write_op.cc:270] Start()
0503 02:12:26.604791 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339072864 usec, L: 0
0503 02:12:26.604967 (+ 176us) write_op.cc:270] Start()
0503 02:12:26.604994 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339092087 usec, L: 0
0503 02:12:26.609660 (+ 4666us) write_op.cc:270] Start()
0503 02:12:26.609705 (+ 45us) write_op.cc:276] Timestamp: P: 1714702339105753 usec, L: 0
0503 02:12:26.609919 (+ 214us) write_op.cc:270] Start()
0503 02:12:26.609945 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339106415 usec, L: 0
0503 02:12:26.610271 (+ 326us) write_op.cc:270] Start()
0503 02:12:26.610298 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339107319 usec, L: 0
0503 02:12:26.610465 (+ 167us) write_op.cc:270] Start()
0503 02:12:26.610491 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339107852 usec, L: 0
0503 02:12:26.610769 (+ 278us) write_op.cc:270] Start()
0503 02:12:26.610799 (+ 30us) write_op.cc:276] Timestamp: P: 1714702339121641 usec, L: 0
0503 02:12:26.610970 (+ 171us) write_op.cc:270] Start()
0503 02:12:26.610996 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339140753 usec, L: 0
0503 02:12:26.611162 (+ 166us) write_op.cc:270] Start()
0503 02:12:26.611189 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339144508 usec, L: 0
0503 02:12:26.611343 (+ 154us) write_op.cc:270] Start()
0503 02:12:26.611367 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339145289 usec, L: 0
0503 02:12:26.611499 (+ 132us) write_op.cc:270] Start()
0503 02:12:26.611516 (+ 17us) write_op.cc:276] Timestamp: P: 1714702339149645 usec, L: 0
0503 02:12:26.611618 (+ 102us) write_op.cc:270] Start()
0503 02:12:26.611681 (+ 63us) write_op.cc:276] Timestamp: P: 1714702339174918 usec, L: 0
0503 02:12:26.611810 (+ 129us) write_op.cc:270] Start()
0503 02:12:26.611836 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339186304 usec, L: 0
0503 02:12:26.612017 (+ 181us) write_op.cc:270] Start()
0503 02:12:26.612039 (+ 22us) write_op.cc:276] Timestamp: P: 1714702339193262 usec, L: 0
0503 02:12:26.612227 (+ 188us) write_op.cc:270] Start()
0503 02:12:26.612250 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339193960 usec, L: 0
0503 02:12:26.612381 (+ 131us) write_op.cc:270] Start()
0503 02:12:26.612401 (+ 20us) write_op.cc:276] Timestamp: P: 1714702339212666 usec, L: 0
0503 02:12:26.612565 (+ 164us) write_op.cc:270] Start()
0503 02:12:26.612589 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339220684 usec, L: 0
0503 02:12:26.612732 (+ 143us) write_op.cc:270] Start()
0503 02:12:26.612756 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339236204 usec, L: 0
0503 02:12:26.612957 (+ 201us) write_op.cc:270] Start()
0503 02:12:26.612987 (+ 30us) write_op.cc:276] Timestamp: P: 1714702339238918 usec, L: 0
0503 02:12:26.613142 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.613167 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339242577 usec, L: 0
0503 02:12:26.613336 (+ 169us) write_op.cc:270] Start()
0503 02:12:26.613360 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339262286 usec, L: 0
0503 02:12:26.613522 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.613546 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339269601 usec, L: 0
0503 02:12:26.613703 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.613727 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339290829 usec, L: 0
0503 02:12:26.613879 (+ 152us) write_op.cc:270] Start()
0503 02:12:26.613903 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339334320 usec, L: 0
0503 02:12:26.614051 (+ 148us) write_op.cc:270] Start()
0503 02:12:26.614076 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339339104 usec, L: 0
0503 02:12:26.614227 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.614251 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339355256 usec, L: 0
0503 02:12:26.616595 (+ 2344us) write_op.cc:270] Start()
0503 02:12:26.616632 (+ 37us) write_op.cc:276] Timestamp: P: 1714702339366018 usec, L: 0
0503 02:12:26.616828 (+ 196us) write_op.cc:270] Start()
0503 02:12:26.616855 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339375260 usec, L: 0
0503 02:12:26.621176 (+ 4321us) write_op.cc:270] Start()
0503 02:12:26.621228 (+ 52us) write_op.cc:276] Timestamp: P: 1714702339378592 usec, L: 0
0503 02:12:26.621448 (+ 220us) write_op.cc:270] Start()
0503 02:12:26.621477 (+ 29us) write_op.cc:276] Timestamp: P: 1714702339382809 usec, L: 0
0503 02:12:26.621657 (+ 180us) write_op.cc:270] Start()
0503 02:12:26.621713 (+ 56us) write_op.cc:276] Timestamp: P: 1714702339385894 usec, L: 0
0503 02:12:26.621885 (+ 172us) write_op.cc:270] Start()
0503 02:12:26.621909 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339391354 usec, L: 0
0503 02:12:26.622066 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.622089 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339396902 usec, L: 0
0503 02:12:26.622266 (+ 177us) write_op.cc:270] Start()
0503 02:12:26.622290 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339429254 usec, L: 0
0503 02:12:26.622453 (+ 163us) write_op.cc:270] Start()
0503 02:12:26.622470 (+ 17us) write_op.cc:276] Timestamp: P: 1714702339445033 usec, L: 0
0503 02:12:26.622634 (+ 164us) write_op.cc:270] Start()
0503 02:12:26.622658 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339452798 usec, L: 0
0503 02:12:26.622816 (+ 158us) write_op.cc:270] Start()
0503 02:12:26.622839 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339501857 usec, L: 0
0503 02:12:26.623095 (+ 256us) write_op.cc:270] Start()
0503 02:12:26.623122 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339515451 usec, L: 0
0503 02:12:26.623301 (+ 179us) write_op.cc:270] Start()
0503 02:12:26.623327 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339522207 usec, L: 0
0503 02:12:26.623490 (+ 163us) write_op.cc:270] Start()
0503 02:12:26.623522 (+ 32us) write_op.cc:276] Timestamp: P: 1714702339540869 usec, L: 0
0503 02:12:26.623682 (+ 160us) write_op.cc:270] Start()
0503 02:12:26.623705 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339559459 usec, L: 0
0503 02:12:26.623867 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.623892 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339582313 usec, L: 0
0503 02:12:26.624047 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.624072 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339595233 usec, L: 0
0503 02:12:26.624229 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.624254 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339606159 usec, L: 0
0503 02:12:26.624398 (+ 144us) write_op.cc:270] Start()
0503 02:12:26.624422 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339613794 usec, L: 0
0503 02:12:26.624587 (+ 165us) write_op.cc:270] Start()
0503 02:12:26.624611 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339721065 usec, L: 0
0503 02:12:26.624772 (+ 161us) write_op.cc:270] Start()
0503 02:12:26.624796 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339743517 usec, L: 0
0503 02:12:26.626099 (+ 1303us) write_op.cc:270] Start()
0503 02:12:26.626141 (+ 42us) write_op.cc:276] Timestamp: P: 1714702339758519 usec, L: 0
0503 02:12:26.626365 (+ 224us) write_op.cc:270] Start()
0503 02:12:26.626392 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339772624 usec, L: 0
0503 02:12:26.626541 (+ 149us) write_op.cc:270] Start()
0503 02:12:26.626567 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339776092 usec, L: 0
0503 02:12:26.626741 (+ 174us) write_op.cc:270] Start()
0503 02:12:26.626765 (+ 24us) write_op.cc:276] Timestamp: P: 1714702339794562 usec, L: 0
0503 02:12:26.631892 (+ 5127us) write_op.cc:270] Start()
0503 02:12:26.631943 (+ 51us) write_op.cc:276] Timestamp: P: 1714702339808198 usec, L: 0
0503 02:12:26.632179 (+ 236us) write_op.cc:270] Start()
0503 02:12:26.632215 (+ 36us) write_op.cc:276] Timestamp: P: 1714702339843598 usec, L: 0
0503 02:12:26.632379 (+ 164us) write_op.cc:270] Start()
0503 02:12:26.632405 (+ 26us) write_op.cc:276] Timestamp: P: 1714702339858663 usec, L: 0
0503 02:12:26.632570 (+ 165us) write_op.cc:270] Start()
0503 02:12:26.632595 (+ 25us) write_op.cc:276] Timestamp: P: 1714702339861502 usec, L: 0
0503 02:12:26.632744 (+ 149us) write_op.cc:270] Start()
0503 02:12:26.632767 (+ 23us) write_op.cc:276] Timestamp: P: 1714702339891310 usec, L: 0
0503 02:12:26.632971 (+ 204us) write_op.cc:270] Start()
0503 02:12:26.632999 (+ 28us) write_op.cc:276] Timestamp: P: 1714702339892928 usec, L: 0
0503 02:12:26.633175 (+ 176us) write_op.cc:270] Start()
0503 02:12:26.633459 (+ 284us) write_op.cc:276] Timestamp: P: 1714702339894405 usec, L: 0
0503 02:12:26.633669 (+ 210us) write_op.cc:270] Start()
0503 02:12:26.633696 (+ 27us) write_op.cc:276] Timestamp: P: 1714702339896500 usec, L: 0
0503 02:12:26.633850 (+ 154us) write_op.cc:270] Start()
0503 02:12:26.634012 (+ 162us) write_op.cc:276] Timestamp: P: 1714702340044251 usec, L: 0
0503 02:12:26.634222 (+ 210us) write_op.cc:270] Start()
0503 02:12:26.634284 (+ 62us) write_op.cc:276] Timestamp: P: 1714702340054739 usec, L: 0
0503 02:12:26.634446 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.634469 (+ 23us) write_op.cc:276] Timestamp: P: 1714702340067746 usec, L: 0
0503 02:12:26.634801 (+ 332us) write_op.cc:270] Start()
0503 02:12:26.634833 (+ 32us) write_op.cc:276] Timestamp: P: 1714702340084478 usec, L: 0
0503 02:12:26.634982 (+ 149us) write_op.cc:270] Start()
0503 02:12:26.635008 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340089577 usec, L: 0
0503 02:12:26.635165 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.635190 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340090238 usec, L: 0
0503 02:12:26.635352 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.635378 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340090964 usec, L: 0
0503 02:12:26.635547 (+ 169us) write_op.cc:270] Start()
0503 02:12:26.635572 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340091539 usec, L: 0
0503 02:12:26.635744 (+ 172us) write_op.cc:270] Start()
0503 02:12:26.635768 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340092017 usec, L: 0
0503 02:12:26.635931 (+ 163us) write_op.cc:270] Start()
0503 02:12:26.635956 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340123612 usec, L: 0
0503 02:12:26.640298 (+ 4342us) write_op.cc:270] Start()
0503 02:12:26.640349 (+ 51us) write_op.cc:276] Timestamp: P: 1714702340124297 usec, L: 0
0503 02:12:26.640583 (+ 234us) write_op.cc:270] Start()
0503 02:12:26.640611 (+ 28us) write_op.cc:276] Timestamp: P: 1714702340124830 usec, L: 0
0503 02:12:26.640757 (+ 146us) write_op.cc:270] Start()
0503 02:12:26.640785 (+ 28us) write_op.cc:276] Timestamp: P: 1714702340125383 usec, L: 0
0503 02:12:26.646908 (+ 6123us) write_op.cc:270] Start()
0503 02:12:26.646973 (+ 65us) write_op.cc:276] Timestamp: P: 1714702340125868 usec, L: 0
0503 02:12:26.647279 (+ 306us) write_op.cc:270] Start()
0503 02:12:26.647308 (+ 29us) write_op.cc:276] Timestamp: P: 1714702340166507 usec, L: 0
0503 02:12:26.647497 (+ 189us) write_op.cc:270] Start()
0503 02:12:26.647521 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340217459 usec, L: 0
0503 02:12:26.647688 (+ 167us) write_op.cc:270] Start()
0503 02:12:26.647714 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340221894 usec, L: 0
0503 02:12:26.647869 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.647897 (+ 28us) write_op.cc:276] Timestamp: P: 1714702340225656 usec, L: 0
0503 02:12:26.648303 (+ 406us) write_op.cc:270] Start()
0503 02:12:26.648341 (+ 38us) write_op.cc:276] Timestamp: P: 1714702340227914 usec, L: 0
0503 02:12:26.648541 (+ 200us) write_op.cc:270] Start()
0503 02:12:26.648567 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340228767 usec, L: 0
0503 02:12:26.648761 (+ 194us) write_op.cc:270] Start()
0503 02:12:26.648786 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340233608 usec, L: 0
0503 02:12:26.649051 (+ 265us) write_op.cc:270] Start()
0503 02:12:26.649078 (+ 27us) write_op.cc:276] Timestamp: P: 1714702340234520 usec, L: 0
0503 02:12:26.649245 (+ 167us) write_op.cc:270] Start()
0503 02:12:26.649270 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340239512 usec, L: 0
0503 02:12:26.649436 (+ 166us) write_op.cc:270] Start()
0503 02:12:26.649491 (+ 55us) write_op.cc:276] Timestamp: P: 1714702340241735 usec, L: 0
0503 02:12:26.649644 (+ 153us) write_op.cc:270] Start()
0503 02:12:26.649669 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340271291 usec, L: 0
0503 02:12:26.649829 (+ 160us) write_op.cc:270] Start()
0503 02:12:26.649855 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340285063 usec, L: 0
0503 02:12:26.650008 (+ 153us) write_op.cc:270] Start()
0503 02:12:26.650034 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340314356 usec, L: 0
0503 02:12:26.650219 (+ 185us) write_op.cc:270] Start()
0503 02:12:26.650244 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340363242 usec, L: 0
0503 02:12:26.650399 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.650423 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340373780 usec, L: 0
0503 02:12:26.650573 (+ 150us) write_op.cc:270] Start()
0503 02:12:26.650599 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340389399 usec, L: 0
0503 02:12:26.650738 (+ 139us) write_op.cc:270] Start()
0503 02:12:26.650762 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340400294 usec, L: 0
0503 02:12:26.650935 (+ 173us) write_op.cc:270] Start()
0503 02:12:26.650960 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340409483 usec, L: 0
0503 02:12:26.651128 (+ 168us) write_op.cc:270] Start()
0503 02:12:26.651152 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340438538 usec, L: 0
0503 02:12:26.651303 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.651329 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340481129 usec, L: 0
0503 02:12:26.651479 (+ 150us) write_op.cc:270] Start()
0503 02:12:26.651503 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340488740 usec, L: 0
0503 02:12:26.651655 (+ 152us) wriI20240503 02:12:28.286341 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet dbfe6b2b9cab4517ae7cabec4796c9ce
I20240503 02:12:28.433583 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:28.818305 4274 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:28.310163 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:28.266098 5611 consensus_queue.cc:1035] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20240503 02:12:28.284500 5114 ts_tablet_manager.cc:1875] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:28.327347 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
te_op.cc:270] Start()
0503 02:12:26.651679 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340491354 usec, L: 0
0503 02:12:26.651830 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.651853 (+ 23us) write_op.cc:276] Timestamp: P: 1714702340503836 usec, L: 0
0503 02:12:26.652163 (+ 310us) write_op.cc:270] Start()
0503 02:12:26.652190 (+ 27us) write_op.cc:276] Timestamp: P: 1714702340505409 usec, L: 0
0503 02:12:26.652385 (+ 195us) write_op.cc:270] Start()
0503 02:12:26.652411 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340507017 usec, L: 0
0503 02:12:26.652590 (+ 179us) write_op.cc:270] Start()
0503 02:12:26.652615 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340508812 usec, L: 0
0503 02:12:26.652794 (+ 179us) write_op.cc:270] Start()
0503 02:12:26.652819 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340545829 usec, L: 0
0503 02:12:26.653018 (+ 199us) write_op.cc:270] Start()
0503 02:12:26.653044 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340570057 usec, L: 0
0503 02:12:26.653211 (+ 167us) write_op.cc:270] Start()
0503 02:12:26.653237 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340598392 usec, L: 0
0503 02:12:26.653428 (+ 191us) write_op.cc:270] Start()
0503 02:12:26.653448 (+ 20us) write_op.cc:276] Timestamp: P: 1714702340773782 usec, L: 0
0503 02:12:26.659546 (+ 6098us) write_op.cc:270] Start()
0503 02:12:26.659620 (+ 74us) write_op.cc:276] Timestamp: P: 1714702340779393 usec, L: 0
0503 02:12:26.659824 (+ 204us) write_op.cc:270] Start()
0503 02:12:26.659852 (+ 28us) write_op.cc:276] Timestamp: P: 1714702340781634 usec, L: 0
0503 02:12:26.660009 (+ 157us) write_op.cc:270] Start()
0503 02:12:26.660035 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340803037 usec, L: 0
0503 02:12:26.660212 (+ 177us) write_op.cc:270] Start()
0503 02:12:26.660240 (+ 28us) write_op.cc:276] Timestamp: P: 1714702340810798 usec, L: 0
0503 02:12:26.660386 (+ 146us) write_op.cc:270] Start()
0503 02:12:26.660412 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340823330 usec, L: 0
0503 02:12:26.660563 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.660599 (+ 36us) write_op.cc:276] Timestamp: P: 1714702340843572 usec, L: 0
0503 02:12:26.660750 (+ 151us) write_op.cc:270] Start()
0503 02:12:26.660777 (+ 27us) write_op.cc:276] Timestamp: P: 1714702340845099 usec, L: 0
0503 02:12:26.660977 (+ 200us) write_op.cc:270] Start()
0503 02:12:26.661004 (+ 27us) write_op.cc:276] Timestamp: P: 1714702340846231 usec, L: 0
0503 02:12:26.661156 (+ 152us) write_op.cc:270] Start()
0503 02:12:26.661179 (+ 23us) write_op.cc:276] Timestamp: P: 1714702340847306 usec, L: 0
0503 02:12:26.661372 (+ 193us) write_op.cc:270] Start()
0503 02:12:26.661396 (+ 24us) write_op.cc:276] Timestamp: P: 1714702340909605 usec, L: 0
0503 02:12:26.661546 (+ 150us) write_op.cc:270] Start()
0503 02:12:26.661571 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340911208 usec, L: 0
0503 02:12:26.661731 (+ 160us) write_op.cc:270] Start()
0503 02:12:26.661789 (+ 58us) write_op.cc:276] Timestamp: P: 1714702340919609 usec, L: 0
0503 02:12:26.661967 (+ 178us) write_op.cc:270] Start()
0503 02:12:26.662179 (+ 212us) write_op.cc:276] Timestamp: P: 1714702340921076 usec, L: 0
0503 02:12:26.662449 (+ 270us) write_op.cc:270] Start()
0503 02:12:26.662483 (+ 34us) write_op.cc:276] Timestamp: P: 1714702340922270 usec, L: 0
0503 02:12:26.662679 (+ 196us) write_op.cc:270] Start()
0503 02:12:26.662705 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340924793 usec, L: 0
0503 02:12:26.662873 (+ 168us) write_op.cc:270] Start()
0503 02:12:26.662931 (+ 58us) write_op.cc:276] Timestamp: P: 1714702340925732 usec, L: 0
0503 02:12:26.663089 (+ 158us) write_op.cc:270] Start()
0503 02:12:26.663110 (+ 21us) write_op.cc:276] Timestamp: P: 1714702340926631 usec, L: 0
0503 02:12:26.663354 (+ 244us) write_op.cc:270] Start()
0503 02:12:26.663379 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340927501 usec, L: 0
0503 02:12:26.663589 (+ 210us) write_op.cc:270] Start()
0503 02:12:26.663614 (+ 25us) write_op.cc:276] Timestamp: P: 1714702340930987 usec, L: 0
0503 02:12:26.663756 (+ 142us) write_op.cc:270] Start()
0503 02:12:26.663776 (+ 20us) write_op.cc:276] Timestamp: P: 1714702340936781 usec, L: 0
0503 02:12:26.663934 (+ 158us) write_op.cc:270] Start()
0503 02:12:26.663957 (+ 23us) write_op.cc:276] Timestamp: P: 1714702340942491 usec, L: 0
0503 02:12:26.664087 (+ 130us) write_op.cc:270] Start()
0503 02:12:26.664108 (+ 21us) write_op.cc:276] Timestamp: P: 1714702340988548 usec, L: 0
0503 02:12:26.664283 (+ 175us) write_op.cc:270] Start()
0503 02:12:26.664310 (+ 27us) write_op.cc:276] Timestamp: P: 1714702340995473 usec, L: 0
0503 02:12:26.664470 (+ 160us) write_op.cc:270] Start()
0503 02:12:26.664496 (+ 26us) write_op.cc:276] Timestamp: P: 1714702340998164 usec, L: 0
0503 02:12:26.664660 (+ 164us) write_op.cc:270] Start()
0503 02:12:26.664684 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341011954 usec, L: 0
0503 02:12:26.664965 (+ 281us) write_op.cc:270] Start()
0503 02:12:26.664993 (+ 28us) write_op.cc:276] Timestamp: P: 1714702341013579 usec, L: 0
0503 02:12:26.665151 (+ 158us) write_op.cc:270] Start()
0503 02:12:26.665175 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341084391 usec, L: 0
0503 02:12:26.665367 (+ 192us) write_op.cc:270] Start()
0503 02:12:26.665393 (+ 26us) write_op.cc:276] Timestamp: P: 1714702341086957 usec, L: 0
0503 02:12:26.665540 (+ 147us) write_op.cc:270] Start()
0503 02:12:26.665563 (+ 23us) write_op.cc:276] Timestamp: P: 1714702341094171 usec, L: 0
0503 02:12:26.665711 (+ 148us) write_op.cc:270] Start()
0503 02:12:26.671002 (+ 5291us) write_op.cc:276] Timestamp: P: 1714702341119076 usec, L: 0
0503 02:12:26.671287 (+ 285us) write_op.cc:270] Start()
0503 02:12:26.671324 (+ 37us) write_op.cc:276] Timestamp: P: 1714702341121062 usec, L: 0
0503 02:12:26.671483 (+ 159us) write_op.cc:270] Start()
0503 02:12:26.671508 (+ 25us) write_op.cc:276] Timestamp: P: 1714702341125105 usec, L: 0
0503 02:12:26.671833 (+ 325us) write_op.cc:270] Start()
0503 02:12:26.671858 (+ 25us) write_op.cc:276] Timestamp: P: 1714702341126320 usec, L: 0
0503 02:12:26.672020 (+ 162us) write_op.cc:270] Start()
0503 02:12:26.672043 (+ 23us) write_op.cc:276] Timestamp: P: 1714702341150036 usec, L: 0
0503 02:12:26.672213 (+ 170us) write_op.cc:270] Start()
0503 02:12:26.672237 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341165806 usec, L: 0
0503 02:12:26.672412 (+ 175us) write_op.cc:270] Start()
0503 02:12:26.672436 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341201544 usec, L: 0
0503 02:12:26.672584 (+ 148us) write_op.cc:270] Start()
0503 02:12:26.672608 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341237431 usec, L: 0
0503 02:12:26.672749 (+ 141us) write_op.cc:270] Start()
0503 02:12:26.672820 (+ 71us) write_op.cc:276] Timestamp: P: 1714702341246573 usec, L: 0
0503 02:12:26.673028 (+ 208us) write_op.cc:270] Start()
0503 02:12:26.673054 (+ 26us) write_op.cc:276] Timestamp: P: 1714702341247555 usec, L: 0
0503 02:12:26.673235 (+ 181us) write_op.cc:270] Start()
0503 02:12:26.673259 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341254692 usec, L: 0
0503 02:12:26.673414 (+ 155us) write_op.cc:270] Start()
0503 02:12:26.673437 (+ 23us) write_op.cc:276] Timestamp: P: 1714702341281997 usec, L: 0
0503 02:12:26.675259 (+ 1822us) write_op.cc:270] Start()
0503 02:12:26.675315 (+ 56us) write_op.cc:276] Timestamp: P: 1714702341340532 usec, L: 0
0503 02:12:26.675549 (+ 234us) write_op.cc:270] Start()
0503 02:12:26.675578 (+ 29us) write_op.cc:276] Timestamp: P: 1714702341366401 usec, L: 0
0503 02:12:26.675745 (+ 167us) write_op.cc:270] Start()
0503 02:12:26.675770 (+ 25us) write_op.cc:276] Timestamp: P: 1714702341388194 usec, L: 0
0503 02:12:26.675929 (+ 159us) write_op.cc:270] Start()
0503 02:12:26.675953 (+ 24us) write_op.cc:276] Timestamp: P: 1714702341398758 usec, L: 0
0503 02:12:26.676093 (+ 140us) write_op.cc:270] Start()
0503 02:12:26.676117 (+ 24us) write_op.cc:276] Timestamp: P: 17147
I20240503 02:12:28.819640 4529 raft_consensus.cc:3037] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.312551 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:28.820398 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6c80efe35343989f46a2c181f38129 (DEFAULT_TABLE table=default.Table_73a988b03f62487b86ace12f89bcd124 [id=65569d68a53e48d0ad75a523117d2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.820839 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6c80efe35343989f46a2c181f38129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.282759 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:28.821483 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 030dc0d3eeb445da91cfeef6886b651d (DEFAULT_TABLE table=default.Table_5c4057da7da24b3e8c9e3e1a4875666e [id=005c67d81b5445deb1e31f613797233c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.287626 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43fc277fa4884c7f92118e333226e800 consensus state has no leader
W20240503 02:12:28.287824 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b32c377006a24372859ed44d59cf90ee consensus state has no leader
I20240503 02:12:28.821899 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030dc0d3eeb445da91cfeef6886b651d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.822306 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2701a2118e9c4a24b91afbb68eebe26c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:28.822387 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_00deed6cfb98406d84258657464b8f93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:28.822932 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00deed6cfb98406d84258657464b8f93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:28.822964 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2701a2118e9c4a24b91afbb68eebe26c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:28.327647 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984fe4640980407aa03c16c14a1303c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.824318 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a915d760ba14a31895a5dc7ec5b34fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:28.276890 5401 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.560s user 0.012s sys 0.000s
I20240503 02:12:28.277072 5159 ts_tablet_manager.cc:1888] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:28.295753 4262 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.007s sys 0.006s
I20240503 02:12:28.825042 5159 log.cc:1196] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/e05d88a5c49b43bea6d46feccaaa6920
W20240503 02:12:28.825129 4262 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 1.165s user 0.007s sys 0.006s
I20240503 02:12:28.825505 5159 ts_tablet_manager.cc:1909] T e05d88a5c49b43bea6d46feccaaa6920 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:28.825840 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 450) took 1818 ms. Trace:
I20240503 02:12:28.825949 5159 rpcz_store.cc:276] 0503 02:12:27.007256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.007307 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:28.180909 (+1173602us) ts_tablet_manager.cc:1120] Deleting tablet e05d88a5c49b43bea6d46feccaaa6920
0503 02:12:28.180936 (+ 27us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.247196 (+ 66260us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.276886 (+ 29690us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.825800 (+548914us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16156,"tablet-delete.queue_time_us":1173301}
I20240503 02:12:28.826370 5159 tablet_replica.cc:330] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:28.826596 5159 raft_consensus.cc:2229] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:28.826795 5159 raft_consensus.cc:2258] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:28.826731 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:28.425427 5420 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "acc6d33969e74c209310005ae911c405" instance_seqno: 1714702335045023) as {username='slave'} at 127.4.17.67:39593; Asking this server to re-register.
W20240503 02:12:28.507629 5400 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.17.124:46689 timed out after 0.738s (SENT)
I20240503 02:12:28.340518 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_873e48ca8c434f55927075d87ac065b1" } modify_external_catalogs: true
I20240503 02:12:28.827366 5159 ts_tablet_manager.cc:1875] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:28.625770 5419 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1f50719d94564844b15e1a9211d48a81" instance_seqno: 1714702334548221) as {username='slave'} at 127.4.17.66:35851; Asking this server to re-register.
I20240503 02:12:28.827380 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_873e48ca8c434f55927075d87ac065b1" } modify_external_catalogs: true
I20240503 02:12:28.345875 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e26f10cba82342b0a8719a897d1af394" } new_table_name: "default.Table_cfe74b06168d471c86ca7d281c6b2121" modify_external_catalogs: true
W20240503 02:12:28.329231 5668 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.000s sys 0.012s
W20240503 02:12:28.827854 5668 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s user 0.000s sys 0.012s
I20240503 02:12:28.333333 5103 raft_consensus.cc:374] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.346376 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_15cc1a53bc594d6b8a04c4588fa6ec14" } new_table_name: "default.Table_75a0bf29850647249a6ed3ad0ca8885e" modify_external_catalogs: true
I20240503 02:12:28.353911 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a7c7733ae249477b9cbd82203830f7a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:28.828056 5103 raft_consensus.cc:729] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.828363 5103 consensus_queue.cc:260] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.828465 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7c7733ae249477b9cbd82203830f7a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:28.363376 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73ec6543b9a468492f3b97772ce43f9 consensus state has no leader
I20240503 02:12:28.364749 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4064ff59de474e0db3a578eae10c3fa6
W20240503 02:12:28.748790 5361 kernel_stack_watchdog.cc:198] Thread 5400 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 103ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa43a84b980 (unknown) at ??:0
@ 0x7fa43a8459c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fa4347cb904 google::LogMessage::Flush() at ??:0
@ 0x7fa4347cba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa43e10bbf4 kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fa43e10b4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa43e0f74e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa43e0f8bd2 kudu::rpc::OutboundCall::SetTimedOut() at ??:0
@ 0x7fa43e090233 kudu::rpc::Connection::HandleOutboundCallTimeout() at ??:0
@ 0x7fa43e08fda6 kudu::rpc::Connection::CallAwaitingResponse::HandleTimeout() at ??:0
@ 0x7fa43e0a7c65 ev::base<>::method_thunk<>() at ??:0
@ 0x7fa435cc8699 ev_invoke_pending at ??:0
@ 0x7fa43e111ba5 kudu::rpc::ReactorThread::InvokePendingCb() at ??:0
@ 0x7fa435cc95a1 ev_run at ??:0
@ 0x7fa43e1217c1 ev::loop_ref::run() at ??:0
I20240503 02:12:28.828879 5103 ts_tablet_manager.cc:1398] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.625s user 0.001s sys 0.001s
I20240503 02:12:28.257385 5105 consensus_queue.cc:260] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.828992 5103 ts_tablet_manager.cc:1430] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Tablet startup took 625ms
I20240503 02:12:28.829372 5105 ts_tablet_manager.cc:1398] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.590s user 0.001s sys 0.002s
W20240503 02:12:28.829126 5103 ts_tablet_manager.cc:1432] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Trace:
0503 02:12:20.426339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.427316 (+ 977us) service_pool.cc:225] Handling call
0503 02:12:20.427681 (+ 365us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.427703 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.444424 (+ 16721us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.506809 (+ 62385us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.510748 (+ 3939us) inbound_call.cc:173] Queueing success response
0503 02:12:27.400633 (+6889885us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:28.203396 (+802763us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:28.208513 (+ 5117us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:28.211126 (+ 2613us) tablet_replica.cc:278] Starting consensus
Metrics: {"tablet-open.queue_time_us":6893726}
I20240503 02:12:28.333585 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c (DEFAULT_TABLE table=default.Table_186402d8f9e2484c89977667d29d1ff9 [id=415ffe34b0784872930d63674d5a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.829473 5105 ts_tablet_manager.cc:1430] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Tablet startup took 589ms
I20240503 02:12:28.337054 4502 tablet_service.cc:1467] Processing CreateTablet for tablet fe20d0ca30f845618e0a20aa85dafbde (DEFAULT_TABLE table=default.Table_252b426637d647128fd41a25217a1cd3 [id=14c5cd8c02fe49bca90a36e1851e7288]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.829576 5105 ts_tablet_manager.cc:1432] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Trace:
0503 02:12:20.218764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.218816 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:20.219377 (+ 561us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.219430 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.226293 (+ 6863us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.272079 (+ 45786us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.273141 (+ 1062us) inbound_call.cc:173] Queueing success response
0503 02:12:28.185323 (+7912182us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:28.239783 (+ 54460us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:28.239987 (+ 204us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:28.240127 (+ 140us) tablet_replica.cc:278] Starting consensus
Metrics: {"tablet-open.queue_time_us":7912208}
I20240503 02:12:28.340314 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 88913ce4890c4f528f47163b6c0f04fe (DEFAULT_TABLE table=default.Table_63c3234b89e94ea4a79400239439762d [id=4b98d85c043d4660919e91032e6f4069]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.350507 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 620204f7cb35442f87eae45f84beeee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.830111 5105 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:28.381600 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0cc2ebfb11aa4840adcc35da73be75a2
I20240503 02:12:28.830322 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 (DEFAULT_TABLE table=default.Table_997fff88119a49a5b51f25324fe4f546 [id=7a6371c00571476eba9ef053d8cba825]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.380983 4462 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.015s sys 0.035s
W20240503 02:12:28.830543 4462 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.194s user 0.016s sys 0.035s
I20240503 02:12:28.830727 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9166690fbabf48e0b6a2cafbbb312ccd" } modify_external_catalogs: true
I20240503 02:12:28.394120 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 335f896171b54b67ad44cb8e161003ce (DEFAULT_TABLE table=default.Table_bcefea9ec04543dc8b079eb3822d739f [id=f171dd35c2d14512b88e39a9023f27ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.831218 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335f896171b54b67ad44cb8e161003ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:28.831403 4462 consensus_peers.cc:485] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:28.831616 4462 leader_election.cc:341] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:28.831815 5105 tablet_bootstrap.cc:654] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.833097 5704 raft_consensus.cc:482] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.833241 5704 raft_consensus.cc:504] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:28.544651 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a73ec6543b9a468492f3b97772ce43f9 (table default.Table_54ab74ddb0d94515895951df323ebd48 [id=eb724fd2c11544ab9d8d539d30a2ffa4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a73ec6543b9a468492f3b97772ce43f9 consensus state has no leader
I20240503 02:12:28.834115 5704 leader_election.cc:288] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:28.410651 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 (DEFAULT_TABLE table=default.Table_e2ed0b0221564739b092ebd62980fc94 [id=4517844c377f4216802514610d506270]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.834679 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552dad16f30a410280c87a6c5ef0eaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.350703 5683 raft_consensus.cc:482] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.317641 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1f8b305ac6424a8bc6a9cf25eec587 (DEFAULT_TABLE table=default.Table_a3621f453c4d4329acee2b8d139c662a [id=8b91af0af2864a4ebcc056e641af86b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.317909 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c (DEFAULT_TABLE table=default.Table_186402d8f9e2484c89977667d29d1ff9 [id=415ffe34b0784872930d63674d5a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.835093 5683 raft_consensus.cc:504] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.835615 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197bca686b5e4d6cb3e6c846ed8eb584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.835801 5683 leader_election.cc:288] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:28.397032 5655 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.011s sys 0.038s
W20240503 02:12:28.836501 5655 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.057s user 0.011s sys 0.039s
I20240503 02:12:28.318404 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 (DEFAULT_TABLE table=default.Table_88f4791ae6054ec5becf0d13aec8be47 [id=0cae1c89bb5048af85cdb5d6ef92489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.396965 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.412741 4791 tablet_service.cc:1467] Processing CreateTablet for tablet b7d0640cea2a482e9974376ba479163b (DEFAULT_TABLE table=default.Table_f3b793f1555b46458c1b7bd8682deeea [id=54e314b5828246988dcf456df7c0876f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.837021 4528 raft_consensus.cc:2454] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:28.320477 4647 tablet_service.cc:1467] Processing CreateTablet for tablet fe20d0ca30f845618e0a20aa85dafbde (DEFAULT_TABLE table=default.Table_252b426637d647128fd41a25217a1cd3 [id=14c5cd8c02fe49bca90a36e1851e7288]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.837361 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d0640cea2a482e9974376ba479163b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:28.397409 5692 log.cc:924] Time spent T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240503 02:12:28.363619 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 8135907b78284d7ca74396cc77dd0612 (DEFAULT_TABLE table=default.Table_3e1aa615a80d47258acd2c493d6cb299 [id=ce9c0a316ba0498781109f1fba39bbd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.368755 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 (DEFAULT_TABLE table=default.Table_997fff88119a49a5b51f25324fe4f546 [id=7a6371c00571476eba9ef053d8cba825]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.428368 5106 tablet_bootstrap.cc:492] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:28.838577 5106 ts_tablet_manager.cc:1367] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 1.355s user 0.006s sys 0.014s
I20240503 02:12:28.839452 5106 raft_consensus.cc:348] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.839705 5106 raft_consensus.cc:374] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.839795 5106 raft_consensus.cc:729] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.840034 5106 consensus_queue.cc:260] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.840502 5106 ts_tablet_manager.cc:1398] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:28.840806 5106 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:28.431586 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4aad131b884c54b69b3d1d9420662c (DEFAULT_TABLE table=default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8 [id=c3eeec6ee1b74545af2b4f26849883f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.414427 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 7900af2ce5034eea9bccdbdf3ea5b709 (DEFAULT_TABLE table=default.Table_e73c3b3756c349dba71457b7d6a40ecf [id=d3a991b1d12241d8b3cdee58fe97e23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.841006 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad17e65b0fc34236a19c7ebd077b75de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.841207 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7900af2ce5034eea9bccdbdf3ea5b709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.438228 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 6b15997a94b74ad3808d965acfb5d155 (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.443200 4670 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.003s sys 0.007s
W20240503 02:12:28.842365 4670 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.986s user 0.003s sys 0.007s
I20240503 02:12:28.457077 4654 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:28.843214 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 (DEFAULT_TABLE table=default.Table_6a0d5b09dfb04a1892fb248b676a923b [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.465637 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 335f896171b54b67ad44cb8e161003ce (DEFAULT_TABLE table=default.Table_bcefea9ec04543dc8b079eb3822d739f [id=f171dd35c2d14512b88e39a9023f27ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.466955 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 43e8fc6409b5448eaa567879c5b7d504 (DEFAULT_TABLE table=default.Table_6d6c3b05fef34441af532f58a3fb5f6a [id=6d141cfe014740069cf37d1d246f2efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.473630 4641 tablet_service.cc:1467] Processing CreateTablet for tablet b7d0640cea2a482e9974376ba479163b (DEFAULT_TABLE table=default.Table_f3b793f1555b46458c1b7bd8682deeea [id=54e314b5828246988dcf456df7c0876f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.478569 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 (DEFAULT_TABLE table=default.Table_e2ed0b0221564739b092ebd62980fc94 [id=4517844c377f4216802514610d506270]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.488937 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 7900af2ce5034eea9bccdbdf3ea5b709 (DEFAULT_TABLE table=default.Table_e73c3b3756c349dba71457b7d6a40ecf [id=d3a991b1d12241d8b3cdee58fe97e23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.496448 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a (DEFAULT_TABLE table=default.Table_8ac84b01b9b94262b9e66dcfb42ded1e [id=e9b3b7cef74c4d85ae8c918af4ef2228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.545516 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 84f6279f16dc42d69e794e3c6586163d (DEFAULT_TABLE table=default.Table_23f94387862b4b6784389cd83a9ac77e [id=70fb3c2002314174a5705603052275df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.546036 4653 tablet_service.cc:1467] Processing CreateTablet for tablet a4b8577a7774488aa768529960b5240d (DEFAULT_TABLE table=default.Table_25ebb22ce7fd40f7bd9828551cc26b6b [id=9526bca19da04a6d8fa0f7842dbdc448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.844769 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b8577a7774488aa768529960b5240d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:28.397606 5589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.600s user 0.020s sys 0.030s
I20240503 02:12:28.845003 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e08c10c15f41878202cef0c8d5ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.549199 5115 ts_tablet_manager.cc:1888] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:28.845136 5115 log.cc:1196] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c0b9003f248b47a9952e6f995887ae0c
I20240503 02:12:28.415951 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a (DEFAULT_TABLE table=default.Table_8ac84b01b9b94262b9e66dcfb42ded1e [id=e9b3b7cef74c4d85ae8c918af4ef2228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.845611 5115 ts_tablet_manager.cc:1909] T c0b9003f248b47a9952e6f995887ae0c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:28.845719 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08e7d17f1644749a9a2e7175b3191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.845954 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 450) took 1263 ms. Trace:
I20240503 02:12:28.846071 5115 rpcz_store.cc:276] 0503 02:12:27.582177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.582682 (+ 505us) service_pool.cc:225] Handling call
0503 02:12:27.583260 (+ 578us) ts_tablet_manager.cc:1120] Deleting tablet c0b9003f248b47a9952e6f995887ae0c
0503 02:12:27.583272 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.461465 (+878193us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.526582 (+ 65117us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.845909 (+319327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":377}
I20240503 02:12:28.846475 5115 tablet_replica.cc:330] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:28.635236 4638 tablet_service.cc:1467] Processing CreateTablet for tablet d2c43838340e44ef92e318a969e34af6 (DEFAULT_TABLE table=default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939 [id=e1a0264ba63544678a88735b11d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.846819 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c43838340e44ef92e318a969e34af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.848569 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5506130048488caa3ee18ff7229835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.640640 5479 consensus_queue.cc:1035] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:28.404680 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 584694f1cc284a718f0e5fa15785d649 (DEFAULT_TABLE table=default.Table_ea8c1983dad340faaa278bfe0d6ea103 [id=b771a5d963eb4ac2b0065061b9feadac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.418345 4787 tablet_service.cc:1467] Processing CreateTablet for tablet a4b8577a7774488aa768529960b5240d (DEFAULT_TABLE table=default.Table_25ebb22ce7fd40f7bd9828551cc26b6b [id=9526bca19da04a6d8fa0f7842dbdc448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.849119 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1f8b305ac6424a8bc6a9cf25eec587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.416754 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1f8b305ac6424a8bc6a9cf25eec587 (DEFAULT_TABLE table=default.Table_a3621f453c4d4329acee2b8d139c662a [id=8b91af0af2864a4ebcc056e641af86b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.418900 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:28.849400 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b8577a7774488aa768529960b5240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.849763 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 8135907b78284d7ca74396cc77dd0612 (DEFAULT_TABLE table=default.Table_3e1aa615a80d47258acd2c493d6cb299 [id=ce9c0a316ba0498781109f1fba39bbd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.419543 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 (DEFAULT_TABLE table=default.Table_88f4791ae6054ec5becf0d13aec8be47 [id=0cae1c89bb5048af85cdb5d6ef92489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.852267 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2141d4c11854a859f1ffdc5ccb953bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.419521 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 84f6279f16dc42d69e794e3c6586163d (DEFAULT_TABLE table=default.Table_23f94387862b4b6784389cd83a9ac77e [id=70fb3c2002314174a5705603052275df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.420990 4774 tablet_service.cc:1467] Processing CreateTablet for tablet d2c43838340e44ef92e318a969e34af6 (DEFAULT_TABLE table=default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939 [id=e1a0264ba63544678a88735b11d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.421973 4790 tablet_service.cc:1467] Processing CreateTablet for tablet b72bff3c50eb4ac6b1b2feb7250ba43e (DEFAULT_TABLE table=default.Table_bb051703036147bd9a71497d00edc307 [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.423061 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6c80efe35343989f46a2c181f38129 (DEFAULT_TABLE table=default.Table_73a988b03f62487b86ace12f89bcd124 [id=65569d68a53e48d0ad75a523117d2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.427862 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 (DEFAULT_TABLE table=default.Table_6a0d5b09dfb04a1892fb248b676a923b [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.433280 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 6e635f283288406aa16b3941be64e777 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.435711 4779 tablet_service.cc:1467] Processing CreateTablet for tablet f3a03a6dc04f47b29401aec463d0d442 (DEFAULT_TABLE table=default.Table_023bb9a5caa9417d9ab87c83b288de66 [id=33ee169586d84b0a95267f799469aeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.437467 4793 tablet_service.cc:1467] Processing CreateTablet for tablet fe35f00dd4e74dc8b6b61eefe2bafaab (DEFAULT_TABLE table=default.Table_a2518b080992412aa7ae7abdd643bdf7 [id=fb33eccba1eb4943ae2bedcf0cd10e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.439321 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 22b0672199fc4d538242c38616f2fd80 (DEFAULT_TABLE table=default.Table_19cd1048ca9d40cb9bfd8f1267a99e0c [id=2c71121bb96e44ffbe0f8d99dbc30d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.445981 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 39c48b14bc624c28a871d45e9429ee30 (DEFAULT_TABLE table=default.Table_b9f90c9e6ae3489b9809bed574c04644 [id=d5ef38cbaed14297b58cf74b9c85244e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.447710 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 (DEFAULT_TABLE table=default.Table_e0f6872c37c746fab51a59df01834e38 [id=ad31fa5a307d4e12a48f892fe41a1d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.449390 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 7099b64f41e54cd7b7967c52892b2200 (DEFAULT_TABLE table=default.Table_923e68dde96e47489191170eb93b19ab [id=f2685456ff4046da8abb2ce85b4a8cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.452059 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 44707b6439b141b7b63f49f34f25b37a (DEFAULT_TABLE table=default.Table_9b6b0e4d8d9f43bf9322382e80b8e842 [id=74f5d88b2f9d4bbe941ebc9fddc18e14]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:28.266707 4746 leader_election.cc:341] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:28.597023 5706 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:28.600517 4735 kernel_stack_watchdog.cc:198] Thread 4746 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 187ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fceda955980 (unknown) at ??:0
@ 0x7fceda94f9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fced48d5904 google::LogMessage::Flush() at ??:0
@ 0x7fced48d5a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fcee2d7c5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fcee2d7f12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fcee2d7eeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fcede2158cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fcede2154ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fcede2014e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fcede201a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fcede19e51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fcede19cf45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fcede1af2c5 ev::base<>::method_thunk<>() at ??:0
W20240503 02:12:28.857703 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 591027us
I20240503 02:12:28.857735 5706 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.858835 5706 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:28.858906 4746 leader_election.cc:302] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:28.641415 4640 tablet_service.cc:1467] Processing CreateTablet for tablet b72bff3c50eb4ac6b1b2feb7250ba43e (DEFAULT_TABLE table=default.Table_bb051703036147bd9a71497d00edc307 [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.859465 5706 raft_consensus.cc:2781] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.859539 5479 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.859613 5706 raft_consensus.cc:482] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.859719 5706 raft_consensus.cc:3037] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.859818 5479 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.860066 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f6279f16dc42d69e794e3c6586163d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.860524 5479 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:28.864552 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c43838340e44ef92e318a969e34af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.869063 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b0672199fc4d538242c38616f2fd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.870498 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.830710 4721 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:28.851999 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c61bf67c8a47fb9ef23bcd6c58c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.873986 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08e7d17f1644749a9a2e7175b3191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.875177 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72bff3c50eb4ac6b1b2feb7250ba43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.877117 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4aad131b884c54b69b3d1d9420662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.881122 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552dad16f30a410280c87a6c5ef0eaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.883913 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868a6ad6cc1a40f091b4516676c24c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.884759 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335f896171b54b67ad44cb8e161003ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.888751 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f6279f16dc42d69e794e3c6586163d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.889065 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d0640cea2a482e9974376ba479163b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.890426 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c919bda1eaaa4e52828aac3ef337f3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.892864 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7900af2ce5034eea9bccdbdf3ea5b709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.893265 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e8fc6409b5448eaa567879c5b7d504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.893831 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a24b5c83b649e79644b08253d851d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.896780 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290a61c21fa456eb3c4f9b0dbebc864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.897176 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8135907b78284d7ca74396cc77dd0612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.900470 4654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.901055 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20d0ca30f845618e0a20aa85dafbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.908191 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b15997a94b74ad3808d965acfb5d155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.914567 5114 ts_tablet_manager.cc:1888] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:28.923280 5419 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:28.851058 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet e05d88a5c49b43bea6d46feccaaa6920 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
W20240503 02:12:28.924783 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 73711.3us
I20240503 02:12:28.925521 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]) successfully deleted
I20240503 02:12:28.921983 5114 log.cc:1196] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c0b9003f248b47a9952e6f995887ae0c
I20240503 02:12:28.911409 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9166690fbabf48e0b6a2cafbbb312ccd" } modify_external_catalogs: true
I20240503 02:12:28.914321 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_064299ed49e74e50a322f6cbf4878b98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:28.916391 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:28.930343 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
W20240503 02:12:28.930603 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064299ed49e74e50a322f6cbf4878b98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:28.934482 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 102622us
I20240503 02:12:28.934562 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:28.936201 4462 leader_election.cc:302] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:28.936857 5668 raft_consensus.cc:2781] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.937023 5668 raft_consensus.cc:482] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.937119 5668 raft_consensus.cc:3037] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.935982 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0bb92f02ee674d1f8317b3ee2e6cebb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:28.938683 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb92f02ee674d1f8317b3ee2e6cebb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:28.938915 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.939136 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.939599 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c18cecefab4b6caed0039d09cc1890" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.939836 4673 raft_consensus.cc:2454] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:28.941016 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:28.943984 5683 raft_consensus.cc:482] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.944211 5683 raft_consensus.cc:504] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:28.945013 5683 leader_election.cc:288] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:28.951598 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c18cecefab4b6caed0039d09cc1890" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.951897 4527 raft_consensus.cc:2454] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:28.952247 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.952447 4528 raft_consensus.cc:2454] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:28.951792 4523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:28.958827 5115 raft_consensus.cc:2229] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:28.959100 5115 raft_consensus.cc:2258] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:28.963729 5115 ts_tablet_manager.cc:1875] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:28.977409 5106 tablet_bootstrap.cc:654] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.988435 4635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 96) took 1192 ms. Trace:
I20240503 02:12:28.988590 4635 rpcz_store.cc:276] 0503 02:12:27.795756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.795962 (+ 206us) service_pool.cc:225] Handling call
0503 02:12:28.961956 (+1165994us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x6160000d0f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:28.988372 (+ 26416us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1377094400}
I20240503 02:12:28.989068 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:28.989389 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:28.819437 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3c2d73cae23744ad8a1b256940d3a750"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:28.991210 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c2d73cae23744ad8a1b256940d3a750 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:29.021585 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6c80efe35343989f46a2c181f38129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.038780 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.039156 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.040318 5736 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:29.037603 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:29.041428 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e635f283288406aa16b3941be64e777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.039513 5706 raft_consensus.cc:504] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:29.024889 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 6e635f283288406aa16b3941be64e777 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.043248 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e635f283288406aa16b3941be64e777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.043380 5115 ts_tablet_manager.cc:1888] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:29.043517 5115 log.cc:1196] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/e05d88a5c49b43bea6d46feccaaa6920
I20240503 02:12:29.043943 5115 ts_tablet_manager.cc:1909] T e05d88a5c49b43bea6d46feccaaa6920 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:29.032192 4648 tablet_service.cc:1467] Processing CreateTablet for tablet f3a03a6dc04f47b29401aec463d0d442 (DEFAULT_TABLE table=default.Table_023bb9a5caa9417d9ab87c83b288de66 [id=33ee169586d84b0a95267f799469aeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.044426 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 452) took 1462 ms. Trace:
I20240503 02:12:29.044517 5115 rpcz_store.cc:276] 0503 02:12:27.582399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.582994 (+ 595us) service_pool.cc:225] Handling call
0503 02:12:28.846409 (+1263415us) ts_tablet_manager.cc:1120] Deleting tablet e05d88a5c49b43bea6d46feccaaa6920
0503 02:12:28.846417 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.967197 (+120780us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.043247 (+ 76050us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.044367 (+ 1120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106626,"tablet-delete.queue_time_us":1263263}
I20240503 02:12:29.034025 4632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 95) took 1342 ms. Trace:
I20240503 02:12:29.044796 4632 rpcz_store.cc:276] 0503 02:12:27.691877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.692450 (+ 573us) service_pool.cc:225] Handling call
0503 02:12:28.965092 (+1272642us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x6160000d0f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:29.033958 (+ 68866us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":981263744}
I20240503 02:12:29.045090 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet e05d88a5c49b43bea6d46feccaaa6920 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
I20240503 02:12:29.046250 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 447) took 1477 ms. Trace:
I20240503 02:12:29.046347 5115 rpcz_store.cc:276] 0503 02:12:27.571874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.571982 (+ 108us) service_pool.cc:225] Handling call
0503 02:12:29.046045 (+1474063us) ts_tablet_manager.cc:1120] Deleting tablet 23ccd25074db40fb96053b915125a425
0503 02:12:29.046055 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.046197 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460854}
W20240503 02:12:29.046798 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 23ccd25074db40fb96053b915125a425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ccd25074db40fb96053b915125a425 already in progress: creating tablet
I20240503 02:12:29.038278 4639 tablet_service.cc:1467] Processing CreateTablet for tablet fe35f00dd4e74dc8b6b61eefe2bafaab (DEFAULT_TABLE table=default.Table_a2518b080992412aa7ae7abdd643bdf7 [id=fb33eccba1eb4943ae2bedcf0cd10e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.052093 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44707b6439b141b7b63f49f34f25b37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.054071 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35f00dd4e74dc8b6b61eefe2bafaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.041254 4631 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 94) took 1455 ms. Trace:
I20240503 02:12:29.054749 4631 rpcz_store.cc:276] 0503 02:12:27.586129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.586648 (+ 519us) service_pool.cc:225] Handling call
0503 02:12:28.964014 (+1377366us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x6160000d0f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:29.041202 (+ 77188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352735616}
I20240503 02:12:29.054847 5706 leader_election.cc:288] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:29.066154 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72bff3c50eb4ac6b1b2feb7250ba43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.067721 5742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:29.067926 5742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:29.069658 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 671 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:29.072299 5114 ts_tablet_manager.cc:1909] T c0b9003f248b47a9952e6f995887ae0c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:29.072744 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290a61c21fa456eb3c4f9b0dbebc864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.076881 4778 tablet_service.cc:1467] Processing CreateTablet for tablet b0e3d3a356f349d0960276be05346888 (DEFAULT_TABLE table=default.Table_ff6cfc8e7a2d4b8a8f691caa77265f87 [id=08935b6cc24d41af8359f67794027b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.077999 5103 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:29.079854 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a03a6dc04f47b29401aec463d0d442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.077906 4499 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 458) took 1424 ms. Trace:
I20240503 02:12:29.083868 4500 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 460) took 1480 ms. Trace:
I20240503 02:12:29.086484 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c48b14bc624c28a871d45e9429ee30. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:29.277063 5361 kernel_stack_watchdog.cc:198] Thread 5400 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 521ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa43a84b980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fa4374b9d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fa4374da7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fa4374d63ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fa4374db4ff kudu::ThreadPool::Submit() at ??:0
@ 0x7fa437402dd4 kudu::DnsResolver::RefreshAddressesAsync() at ??:0
@ 0x7fa43e108b4a kudu::rpc::Proxy::RefreshDnsAndEnqueueRequest() at ??:0
@ 0x7fa43e10bd04 kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fa43e10b4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa43e0f74e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa43e0f8bd2 kudu::rpc::OutboundCall::SetTimedOut() at ??:0
@ 0x7fa43e090233 kudu::rpc::Connection::HandleOutboundCallTimeout() at ??:0
I20240503 02:12:29.087404 4495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 454) took 1539 ms. Trace:
I20240503 02:12:29.058370 4633 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 93) took 1489 ms. Trace:
I20240503 02:12:29.087567 5742 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:29.770771 4495 rpcz_store.cc:276] 0503 02:12:27.534572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.534637 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:27.535110 (+ 473us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.535137 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.831776 (+1296639us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.834462 (+ 2686us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.074356 (+239894us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.770865 4633 rpcz_store.cc:276] 0503 02:12:27.569038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.579641 (+ 10603us) service_pool.cc:225] Handling call
0503 02:12:28.114304 (+534663us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x616000124a30. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9aa2f628 00007f1c9d42d988 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:28.969116 (+854812us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x6160000d0f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:29.057984 (+ 88868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1259326592}
I20240503 02:12:29.771498 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 (DEFAULT_TABLE table=default.Table_e2ed0b0221564739b092ebd62980fc94 [id=4517844c377f4216802514610d506270]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.087991 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4aad131b884c54b69b3d1d9420662c (DEFAULT_TABLE table=default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8 [id=c3eeec6ee1b74545af2b4f26849883f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.089555 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:29.092547 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584694f1cc284a718f0e5fa15785d649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.276827 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:29.772589 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88913ce4890c4f528f47163b6c0f04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.099162 4670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.4.17.67:35679 (request call id 112) took 1251 ms. Trace:
I20240503 02:12:29.087044 5159 ts_tablet_manager.cc:1888] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:29.281077 5400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 773418us
I20240503 02:12:29.093580 4499 rpcz_store.cc:276] 0503 02:12:27.535826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.537499 (+ 1673us) service_pool.cc:225] Handling call
0503 02:12:27.537838 (+ 339us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.537861 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.887895 (+1350034us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.956766 (+ 68871us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:28.960289 (+ 3523us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:29.662912 4330 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.17.126:43321 for RPC RequestConsensusVote timed out after 0.590s (ON_OUTBOUND_QUEUE) [suppressed 1 similar messages]
I20240503 02:12:29.773579 4499 tablet_service.cc:1467] Processing CreateTablet for tablet b7d0640cea2a482e9974376ba479163b (DEFAULT_TABLE table=default.Table_f3b793f1555b46458c1b7bd8682deeea [id=54e314b5828246988dcf456df7c0876f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.467543 4267 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 683) took 1183 ms. Trace:
I20240503 02:12:29.774056 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d0640cea2a482e9974376ba479163b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.773854 4267 rpcz_store.cc:276] 0503 02:12:28.284278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.822065 (+537787us) service_pool.cc:225] Handling call
0503 02:12:28.823068 (+ 1003us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.823629 (+ 561us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.467200 (+643571us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.467351 (+ 151us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.467491 (+ 140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:28.969256 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:28.969402 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:28.969409 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:28.969412 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:28.969528 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:28.969544 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:28.969550 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:28.969652 (+ 102us) tablet.cc:699] Row locks acquired
0503 02:12:28.969654 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:28.969699 (+ 45us) write_op.cc:270] Start()
0503 02:12:28.969740 (+ 41us) write_op.cc:276] Timestamp: P: 1714702348969584 usec, L: 0
0503 02:12:28.969744 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:28.969986 (+ 242us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.334572 (+364586us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.419204 (+ 84632us) write_op.cc:301] APPLY: starting
0503 02:12:29.421433 (+ 2229us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.427209 (+ 5776us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.430446 (+ 3237us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.448099 (+ 17653us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.449963 (+ 1864us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.450605 (+ 642us) write_op.cc:312] APPLY: finished
0503 02:12:29.456724 (+ 6119us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.462363 (+ 5639us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.464239 (+ 1876us) write_op.cc:454] Released schema lock
0503 02:12:29.464976 (+ 737us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.470418 (+ 5442us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.471053 (+ 635us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":62976,"child_traces":[["op",{"apply.queue_time_us":78879,"apply.run_cpu_time_us":5189,"apply.run_wall_time_us":58400,"num_ops":2,"prepare.queue_time_us":145157,"prepare.run_cpu_time_us":10306,"prepare.run_wall_time_us":253864,"raft.queue_time_us":118227,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":495,"replication_time_us":364804,"thread_start_us":357441,"threads_started":2}]]}
I20240503 02:12:29.100544 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099b64f41e54cd7b7967c52892b2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.776978 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c61bf67c8a47fb9ef23bcd6c58c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.097261 4634 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 97) took 1055 ms. Trace:
I20240503 02:12:29.777338 4634 rpcz_store.cc:276] 0503 02:12:28.042015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.042080 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:28.973117 (+931037us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x6160000d0f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:29.097212 (+124095us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1642504576}
I20240503 02:12:29.661110 5688 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:29.778801 5688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: loading cluster ID for follower catalog manager: success
I20240503 02:12:29.108697 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.112388 5736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:39380 (request call id 485) took 1279 ms. Trace:
I20240503 02:12:29.779920 5736 rpcz_store.cc:276] 0503 02:12:27.832731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.837433 (+ 4702us) service_pool.cc:225] Handling call
0503 02:12:29.112313 (+1274880us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.837968 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:27.838143 (+ 175us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:27.838152 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:27.838158 (+ 6us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:27.838217 (+ 59us) alter_schema_op.cc:119] START. Timestamp: 1714702347838061
0503 02:12:27.838220 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.838433 (+ 213us) log.cc:842] Serialized 173 byte log entry
0503 02:12:28.889619 (+1051186us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.040241 (+150622us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:29.081074 (+ 40833us) log.cc:842] Serialized 5 byte log entry
0503 02:12:29.111854 (+ 30780us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.112214 (+ 360us) log.cc:842] Serialized 14 byte log entry
0503 02:12:29.112273 (+ 59us) alter_schema_op.cc:65] Released schema lock
0503 02:12:29.112280 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":150506,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":30715,"prepare.run_wall_time_us":307350,"replication_time_us":1051365,"thread_start_us":440819,"threads_started":2,"wal-append.queue_time_us":294736}]]}
I20240503 02:12:29.114715 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.529923 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet dedead5a2bf347f6991cce8e9b4db708
I20240503 02:12:29.092623 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 6b15997a94b74ad3808d965acfb5d155 (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.118872 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.123339 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b60837d0810041cc883d3d8056d11bb5 (DEFAULT_TABLE table=default.Table_e57bbe28945b49eb9a52ef6f4170ca13 [id=669f20def97e41449baf70a130d948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.781352 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b15997a94b74ad3808d965acfb5d155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.781553 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60837d0810041cc883d3d8056d11bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.781553 4271 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 679) took 1547 ms. Trace:
I20240503 02:12:29.101693 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 22b0672199fc4d538242c38616f2fd80 (DEFAULT_TABLE table=default.Table_19cd1048ca9d40cb9bfd8f1267a99e0c [id=2c71121bb96e44ffbe0f8d99dbc30d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.123512 4787 tablet_service.cc:1467] Processing CreateTablet for tablet d40ed707ad074e0fbb0ee48c2137c661 (DEFAULT_TABLE table=default.Table_081e142b6f9e45b5b4cb3987b8f31ce1 [id=3829412f782f4f8ca43a8d851c0ec4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.772853 5159 log.cc:1196] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/79884b3e73904c708d0d48bca85351b7
I20240503 02:12:29.094146 4529 raft_consensus.cc:2454] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:29.785297 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40ed707ad074e0fbb0ee48c2137c661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.785367 5159 ts_tablet_manager.cc:1909] T 79884b3e73904c708d0d48bca85351b7 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:29.785405 4529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.67:34503 (request call id 114) took 1502 ms. Trace:
I20240503 02:12:29.785565 4529 rpcz_store.cc:276] 0503 02:12:28.283403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.283468 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:29.785348 (+1501880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.108781 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a03a6dc04f47b29401aec463d0d442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.786223 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 449) took 2779 ms. Trace:
I20240503 02:12:29.786338 5159 rpcz_store.cc:276] 0503 02:12:27.006628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.034346 (+ 27718us) service_pool.cc:225] Handling call
0503 02:12:28.826305 (+1791959us) ts_tablet_manager.cc:1120] Deleting tablet 79884b3e73904c708d0d48bca85351b7
0503 02:12:28.826316 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.057670 (+231354us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.086928 (+ 29258us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.786149 (+699221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770291}
I20240503 02:12:29.789176 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "145afaa7548f4b7fafb23f20276f0cb4" is_pre_election: true
I20240503 02:12:29.846841 5688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: acquiring CA information for follower catalog manager: success
I20240503 02:12:29.128193 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 7099b64f41e54cd7b7967c52892b2200 (DEFAULT_TABLE table=default.Table_923e68dde96e47489191170eb93b19ab [id=f2685456ff4046da8abb2ce85b4a8cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.095345 4500 rpcz_store.cc:276] 0503 02:12:27.593342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.593741 (+ 399us) service_pool.cc:225] Handling call
0503 02:12:27.623526 (+ 29785us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.623551 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.837869 (+1214318us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.840731 (+ 2862us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.074100 (+233369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.097679 4493 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 457) took 1561 ms. Trace:
I20240503 02:12:29.953752 4493 rpcz_store.cc:276] 0503 02:12:27.535059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.537249 (+ 2190us) service_pool.cc:225] Handling call
0503 02:12:27.537602 (+ 353us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.537622 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.897925 (+1360303us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.096708 (+198783us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.096885 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.954151 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7900af2ce5034eea9bccdbdf3ea5b709 (DEFAULT_TABLE table=default.Table_e73c3b3756c349dba71457b7d6a40ecf [id=d3a991b1d12241d8b3cdee58fe97e23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.954322 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a (DEFAULT_TABLE table=default.Table_8ac84b01b9b94262b9e66dcfb42ded1e [id=e9b3b7cef74c4d85ae8c918af4ef2228]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.781673 4271 rpcz_store.cc:276] 0503 02:12:28.234446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.285204 (+ 50758us) service_pool.cc:225] Handling call
0503 02:12:28.340405 (+ 55201us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:28.827530 (+487125us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:28.827697 (+ 167us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:28.827775 (+ 78us) catalog_manager.cc:2758] Locking tablets
0503 02:12:28.827937 (+ 162us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:29.529636 (+701699us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:29.529769 (+ 133us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:29.529834 (+ 65us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:29.781497 (+251663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.224030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.226055 (+ 2025us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.251720 (+ 25665us) write_op.cc:435] Acquired schema lock
0503 02:12:29.251730 (+ 10us) tablet.cc:660] Decoding operations
0503 02:12:29.256003 (+ 4273us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.256503 (+ 500us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.256748 (+ 245us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.305437 (+ 48689us) tablet.cc:699] Row locks acquired
0503 02:12:29.305506 (+ 69us) write_op.cc:260] PREPARE: finished
0503 02:12:29.305747 (+ 241us) write_op.cc:270] Start()
0503 02:12:29.305897 (+ 150us) write_op.cc:276] Timestamp: P: 1714702349305663 usec, L: 0
0503 02:12:29.305953 (+ 56us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.320982 (+ 15029us) log.cc:842] Serialized 790 byte log entry
0503 02:12:29.527483 (+206501us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.528214 (+ 731us) write_op.cc:301] APPLY: starting
0503 02:12:29.528269 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.528358 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.528487 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.528672 (+ 185us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.528752 (+ 80us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:29.528765 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:29.528981 (+ 216us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.529402 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.529458 (+ 56us) write_op.cc:454] Released schema lock
0503 02:12:29.529469 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.529595 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.529630 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":126215552,"child_traces":[["op",{"apply.queue_time_us":627,"apply.run_cpu_time_us":972,"apply.run_wall_time_us":1523,"num_ops":2,"prepare.queue_time_us":394939,"prepare.run_cpu_time_us":4973,"prepare.run_wall_time_us":99628,"replication_time_us":221464}]]}
I20240503 02:12:29.547425 4266 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 681) took 1282 ms. Trace:
I20240503 02:12:29.091835 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 43e8fc6409b5448eaa567879c5b7d504 (DEFAULT_TABLE table=default.Table_6d6c3b05fef34441af532f58a3fb5f6a [id=6d141cfe014740069cf37d1d246f2efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.168813 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:29.955107 4497 tablet_service.cc:1467] Processing CreateTablet for tablet a4b8577a7774488aa768529960b5240d (DEFAULT_TABLE table=default.Table_25ebb22ce7fd40f7bd9828551cc26b6b [id=9526bca19da04a6d8fa0f7842dbdc448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.954708 4266 rpcz_store.cc:276] 0503 02:12:28.265118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.353618 (+ 88500us) service_pool.cc:225] Handling call
0503 02:12:28.911583 (+557965us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.912199 (+ 616us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.546958 (+634759us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.547108 (+ 150us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.547241 (+ 133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.331336 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.332303 (+ 967us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.332310 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:29.332432 (+ 122us) tablet.cc:660] Decoding operations
0503 02:12:29.334444 (+ 2012us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.334521 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.334575 (+ 54us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.334804 (+ 229us) tablet.cc:699] Row locks acquired
0503 02:12:29.334807 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:29.334940 (+ 133us) write_op.cc:270] Start()
0503 02:12:29.335072 (+ 132us) write_op.cc:276] Timestamp: P: 1714702349334866 usec, L: 0
0503 02:12:29.335077 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.440352 (+105275us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.440570 (+ 218us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:29.534893 (+ 94323us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.540819 (+ 5926us) write_op.cc:301] APPLY: starting
0503 02:12:29.540843 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.541684 (+ 841us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.541688 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.544578 (+ 2890us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.544965 (+ 387us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.544978 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:29.546248 (+ 1270us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.546754 (+ 506us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.546784 (+ 30us) write_op.cc:454] Released schema lock
0503 02:12:29.546821 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.546961 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.546966 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":837,"apply.run_cpu_time_us":1072,"apply.run_wall_time_us":7260,"num_ops":2,"prepare.queue_time_us":418499,"prepare.run_cpu_time_us":3214,"prepare.run_wall_time_us":109395,"replication_time_us":199787,"spinlock_wait_cycles":241549952}]]}
I20240503 02:12:29.171933 5668 raft_consensus.cc:504] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:29.786800 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 451) took 2748 ms. Trace:
I20240503 02:12:29.955952 5668 leader_election.cc:288] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:29.203189 4496 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 453) took 1541 ms. Trace:
I20240503 02:12:29.955897 5159 rpcz_store.cc:276] 0503 02:12:27.037943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.057912 (+ 19969us) service_pool.cc:225] Handling call
0503 02:12:29.786642 (+2728730us) ts_tablet_manager.cc:1120] Deleting tablet 2a51563396614c54b980f0679e84a2b1
0503 02:12:29.786651 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.786743 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2649354}
I20240503 02:12:29.956125 4496 rpcz_store.cc:276] 0503 02:12:27.534440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.535110 (+ 670us) service_pool.cc:225] Handling call
0503 02:12:27.535527 (+ 417us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.535554 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.847784 (+1312230us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.939009 (+ 91225us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.076040 (+137031us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:29.552942 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfcc73e2eb57429aa61b52b42198c747 consensus state has no leader
I20240503 02:12:29.956516 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 458) took 2593 ms. Trace:
I20240503 02:12:29.956557 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 84f6279f16dc42d69e794e3c6586163d (DEFAULT_TABLE table=default.Table_23f94387862b4b6784389cd83a9ac77e [id=70fb3c2002314174a5705603052275df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.956684 4270 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 680) took 1699 ms. Trace:
I20240503 02:12:29.956612 5159 rpcz_store.cc:276] 0503 02:12:27.363479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.363999 (+ 520us) service_pool.cc:225] Handling call
0503 02:12:29.956319 (+2592320us) ts_tablet_manager.cc:1120] Deleting tablet 3b72c20a4b5a4d2ea266f031733d1aab
0503 02:12:29.956332 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.956456 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2585974}
I20240503 02:12:29.206916 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 335f896171b54b67ad44cb8e161003ce (DEFAULT_TABLE table=default.Table_bcefea9ec04543dc8b079eb3822d739f [id=f171dd35c2d14512b88e39a9023f27ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.957024 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 460) took 2524 ms. Trace:
I20240503 02:12:29.957115 5159 rpcz_store.cc:276] 0503 02:12:27.432361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.432415 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:29.956836 (+2524421us) ts_tablet_manager.cc:1120] Deleting tablet d39f042afb954daa923502ef12377bd5
0503 02:12:29.956845 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.956970 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524175}
I20240503 02:12:29.957448 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 459) took 2526 ms. Trace:
I20240503 02:12:29.957540 5159 rpcz_store.cc:276] 0503 02:12:27.431382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.432711 (+ 1329us) service_pool.cc:225] Handling call
0503 02:12:29.957301 (+2524590us) ts_tablet_manager.cc:1120] Deleting tablet 23ccd25074db40fb96053b915125a425
0503 02:12:29.957309 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.957400 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524422}
I20240503 02:12:29.957814 5159 tablet_replica.cc:330] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:29.957996 5159 raft_consensus.cc:2229] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:29.958191 5159 raft_consensus.cc:2258] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:29.958786 5159 ts_tablet_manager.cc:1875] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:29.816375 4330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 153452us
I20240503 02:12:29.131763 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 44707b6439b141b7b63f49f34f25b37a (DEFAULT_TABLE table=default.Table_9b6b0e4d8d9f43bf9322382e80b8e842 [id=74f5d88b2f9d4bbe941ebc9fddc18e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.129343 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 (DEFAULT_TABLE table=default.Table_b3dcd45817264953aa80592e5bd4c4e4 [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:29.553273 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b9470480a9de45209044f8e8c461519a (table default.Table_9313c394ecf4460a81267e309fdd6748 [id=838900269aa04721a4033aab3345c5dc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b9470480a9de45209044f8e8c461519a consensus state has no leader
I20240503 02:12:29.212679 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 448) took 1769 ms. Trace:
I20240503 02:12:29.960476 5400 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c621d795c44d0daa534261920d874a; no voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:29.960635 5114 rpcz_store.cc:276] 0503 02:12:27.442701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.442762 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:27.736016 (+293254us) ts_tablet_manager.cc:1120] Deleting tablet c0b9003f248b47a9952e6f995887ae0c
0503 02:12:27.736029 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:28.844338 (+1108309us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.902453 (+ 58115us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.212634 (+310181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7472,"tablet-delete.queue_time_us":293026}
I20240503 02:12:29.961087 5114 tablet_replica.cc:330] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:29.961326 5114 raft_consensus.cc:2229] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:29.961522 5114 raft_consensus.cc:2258] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:29.227677 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet e05d88a5c49b43bea6d46feccaaa6920 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.962139 5114 ts_tablet_manager.cc:1875] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:29.961969 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 671 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:29.560748 4275 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 686) took 1063 ms. Trace:
I20240503 02:12:29.962654 4275 rpcz_store.cc:276] 0503 02:12:28.497259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.914049 (+416790us) service_pool.cc:225] Handling call
0503 02:12:28.930728 (+ 16679us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.931289 (+ 561us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.560407 (+629118us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.560563 (+ 156us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.560683 (+ 120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.441309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.441401 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.441405 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:29.441407 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:29.441495 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.441504 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.441508 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.441570 (+ 62us) tablet.cc:699] Row locks acquired
0503 02:12:29.441572 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:29.441602 (+ 30us) write_op.cc:270] Start()
0503 02:12:29.441625 (+ 23us) write_op.cc:276] Timestamp: P: 1714702349441569 usec, L: 0
0503 02:12:29.441628 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.441763 (+ 135us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.535003 (+ 93240us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.552011 (+ 17008us) write_op.cc:301] APPLY: starting
0503 02:12:29.552350 (+ 339us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.552709 (+ 359us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.553601 (+ 892us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.555598 (+ 1997us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.555949 (+ 351us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.555958 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:29.558130 (+ 2172us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.558863 (+ 733us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.559895 (+ 1032us) write_op.cc:454] Released schema lock
0503 02:12:29.559904 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.561277 (+ 1373us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.561283 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15885,"apply.run_cpu_time_us":1128,"apply.run_wall_time_us":10465,"num_ops":2,"prepare.queue_time_us":509479,"prepare.run_cpu_time_us":500,"prepare.run_wall_time_us":500,"replication_time_us":93363,"thread_start_us":13788,"threads_started":1}]]}
I20240503 02:12:29.142807 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 030dc0d3eeb445da91cfeef6886b651d (DEFAULT_TABLE table=default.Table_5c4057da7da24b3e8c9e3e1a4875666e [id=005c67d81b5445deb1e31f613797233c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.213867 4494 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 471) took 1065 ms. Trace:
I20240503 02:12:29.963778 4494 rpcz_store.cc:276] 0503 02:12:28.148528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.148611 (+ 83us) service_pool.cc:225] Handling call
0503 02:12:28.161203 (+ 12592us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:28.161241 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.902193 (+740952us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:28.959190 (+ 56997us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.213555 (+254365us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.964284 4494 tablet_service.cc:1467] Processing CreateTablet for tablet d2c43838340e44ef92e318a969e34af6 (DEFAULT_TABLE table=default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939 [id=e1a0264ba63544678a88735b11d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.227805 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.575263 4268 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 687) took 1077 ms. Trace:
I20240503 02:12:29.230788 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:29.250916 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 391735us
I20240503 02:12:29.145514 4638 tablet_service.cc:1467] Processing CreateTablet for tablet b0e3d3a356f349d0960276be05346888 (DEFAULT_TABLE table=default.Table_ff6cfc8e7a2d4b8a8f691caa77265f87 [id=08935b6cc24d41af8359f67794027b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.149502 4643 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1a5d3ddc24b44b921648bd94ad827 (DEFAULT_TABLE table=default.Table_01fcdbe2fef14f7284c1f68c4f72e92c [id=29ccc9b87d064083ae19ae549e3db5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.155572 4651 tablet_service.cc:1467] Processing CreateTablet for tablet d40ed707ad074e0fbb0ee48c2137c661 (DEFAULT_TABLE table=default.Table_081e142b6f9e45b5b4cb3987b8f31ce1 [id=3829412f782f4f8ca43a8d851c0ec4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.155989 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 (DEFAULT_TABLE table=default.Table_b3dcd45817264953aa80592e5bd4c4e4 [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.968595 4746 leader_election.cc:302] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:29.237813 4501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 459) took 1648 ms. Trace:
I20240503 02:12:29.293819 5105 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:29.198778 4636 tablet_service.cc:1467] Processing CreateTablet for tablet b60837d0810041cc883d3d8056d11bb5 (DEFAULT_TABLE table=default.Table_e57bbe28945b49eb9a52ef6f4170ca13 [id=669f20def97e41449baf70a130d948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.965551 4268 rpcz_store.cc:276] 0503 02:12:28.497341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.935720 (+438379us) service_pool.cc:225] Handling call
0503 02:12:28.938796 (+ 3076us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.939318 (+ 522us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.574737 (+635419us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.574905 (+ 168us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.575043 (+ 138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.441838 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.441943 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.441951 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:29.441954 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:29.442029 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.442038 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.442043 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.442105 (+ 62us) tablet.cc:699] Row locks acquired
0503 02:12:29.442107 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:29.442135 (+ 28us) write_op.cc:270] Start()
0503 02:12:29.442161 (+ 26us) write_op.cc:276] Timestamp: P: 1714702349442103 usec, L: 0
0503 02:12:29.442165 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.442292 (+ 127us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.549282 (+106990us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.554524 (+ 5242us) write_op.cc:301] APPLY: starting
0503 02:12:29.554767 (+ 243us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.556181 (+ 1414us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.557975 (+ 1794us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.563573 (+ 5598us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.564881 (+ 1308us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.565215 (+ 334us) write_op.cc:312] APPLY: finished
0503 02:12:29.569149 (+ 3934us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.571808 (+ 2659us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.573129 (+ 1321us) write_op.cc:454] Released schema lock
0503 02:12:29.573580 (+ 451us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.575760 (+ 2180us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.576067 (+ 307us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3913,"apply.run_cpu_time_us":3650,"apply.run_wall_time_us":24175,"num_ops":2,"prepare.queue_time_us":501993,"prepare.run_cpu_time_us":501,"prepare.run_wall_time_us":500,"replication_time_us":106900,"thread_start_us":29167,"threads_started":1}]]}
I20240503 02:12:30.190261 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b12b7f4b0c7048f38687ebc605d55be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:30.190991 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b12b7f4b0c7048f38687ebc605d55be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:30.015062 5400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 54444.8us
I20240503 02:12:29.968837 4501 rpcz_store.cc:276] 0503 02:12:27.589168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.589333 (+ 165us) service_pool.cc:225] Handling call
0503 02:12:27.623234 (+ 33901us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.623276 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.883385 (+1260109us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.236255 (+352870us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.237411 (+ 1156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.584362 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3478033c4ed94268a2436a2942c27a68
I20240503 02:12:29.594875 4272 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 684) took 1301 ms. Trace:
I20240503 02:12:30.192909 4501 tablet_service.cc:1467] Processing CreateTablet for tablet b72bff3c50eb4ac6b1b2feb7250ba43e (DEFAULT_TABLE table=default.Table_bb051703036147bd9a71497d00edc307 [id=07ffc27bc99f43a8b690d59638e737f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.192615 4272 rpcz_store.cc:276] 0503 02:12:28.293105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.822076 (+528971us) service_pool.cc:225] Handling call
0503 02:12:28.823026 (+ 950us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.823579 (+ 553us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.594528 (+770949us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.594676 (+ 148us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.594800 (+ 124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.442384 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.442460 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.442466 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:29.442468 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:29.442536 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.442544 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.442549 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.442610 (+ 61us) tablet.cc:699] Row locks acquired
0503 02:12:29.442612 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:29.442639 (+ 27us) write_op.cc:270] Start()
0503 02:12:29.442662 (+ 23us) write_op.cc:276] Timestamp: P: 1714702349442606 usec, L: 0
0503 02:12:29.442666 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.442812 (+ 146us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.579222 (+136410us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.582867 (+ 3645us) write_op.cc:301] APPLY: starting
0503 02:12:29.582884 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.590563 (+ 7679us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.590632 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.590781 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.590826 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.590836 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:29.590960 (+ 124us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.593693 (+ 2733us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.593721 (+ 28us) write_op.cc:454] Released schema lock
0503 02:12:29.594024 (+ 303us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.595534 (+ 1510us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.595541 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":60800,"child_traces":[["op",{"apply.queue_time_us":1418,"apply.run_cpu_time_us":945,"apply.run_wall_time_us":13335,"mutex_wait_us":145413,"num_ops":2,"prepare.queue_time_us":618289,"prepare.run_cpu_time_us":483,"prepare.run_wall_time_us":483,"replication_time_us":136391}]]}
I20240503 02:12:29.969115 5105 ts_tablet_manager.cc:1367] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 1.139s user 0.006s sys 0.006s
I20240503 02:12:30.193665 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1e4923fbee8e4eb59344e61bfd2b499f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:30.192992 5688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240503 02:12:30.194298 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e4923fbee8e4eb59344e61bfd2b499f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:30.194281 5105 raft_consensus.cc:348] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.194564 5105 raft_consensus.cc:374] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.194641 5105 raft_consensus.cc:729] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.194902 5105 consensus_queue.cc:260] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:29.202040 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:30.017292 5790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240503 02:12:29.178239 4462 leader_election.cc:341] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:29.524870 4447 kernel_stack_watchdog.cc:198] Thread 4462 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 275ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f13966d2980 (unknown) at ??:0
@ 0x7f13966cc9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f1390652904 google::LogMessage::Flush() at ??:0
@ 0x7f1390652a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f139eaf95d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f139eafc12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f139eafbeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1399f928cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f1399f924ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1399f7e4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f1399f7ea2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f1399f1b51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f1399f19f45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f1399f2c2c5 ev::base<>::method_thunk<>() at ??:0
W20240503 02:12:30.195649 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 1.10256e+06us
I20240503 02:12:29.790607 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4aad131b884c54b69b3d1d9420662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.819988 4503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 491) took 1484 ms. Trace:
W20240503 02:12:30.196409 4462 leader_election.cc:341] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:29.818881 4491 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 481) took 1593 ms. Trace:
W20240503 02:12:29.531049 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a86ac17982324847949b8a9245329489 consensus state has no leader
I20240503 02:12:29.581990 4274 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 682) took 1307 ms. Trace:
I20240503 02:12:30.196590 4491 rpcz_store.cc:276] 0503 02:12:28.224397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.224499 (+ 102us) service_pool.cc:225] Handling call
0503 02:12:29.075829 (+851330us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.075877 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.776022 (+700145us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.812832 (+ 36810us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.817289 (+ 4457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.956794 4270 rpcz_store.cc:276] 0503 02:12:28.257627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.286644 (+ 29017us) service_pool.cc:225] Handling call
0503 02:12:28.346317 (+ 59673us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:28.912764 (+566447us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:28.913045 (+ 281us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:28.913269 (+ 224us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:28.913357 (+ 88us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:29.552606 (+639249us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:29.956643 (+404037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.440651 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.440763 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.440769 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:29.440772 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:29.440874 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.440885 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.440889 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:29.440970 (+ 81us) tablet.cc:699] Row locks acquired
0503 02:12:29.440973 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:29.441012 (+ 39us) write_op.cc:270] Start()
0503 02:12:29.441045 (+ 33us) write_op.cc:276] Timestamp: P: 1714702349440976 usec, L: 0
0503 02:12:29.441051 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.441218 (+ 167us) log.cc:842] Serialized 405 byte log entry
0503 02:12:29.534970 (+ 93752us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.540669 (+ 5699us) write_op.cc:301] APPLY: starting
0503 02:12:29.541064 (+ 395us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.541834 (+ 770us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.542627 (+ 793us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.543230 (+ 603us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.544103 (+ 873us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:29.544444 (+ 341us) write_op.cc:312] APPLY: finished
0503 02:12:29.547431 (+ 2987us) log.cc:842] Serialized 23 byte log entry
0503 02:12:29.551382 (+ 3951us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.551621 (+ 239us) write_op.cc:454] Released schema lock
0503 02:12:29.551851 (+ 230us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.553422 (+ 1571us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.553655 (+ 233us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":133183232,"child_traces":[["op",{"apply.queue_time_us":4992,"apply.run_cpu_time_us":2337,"apply.run_wall_time_us":14470,"num_ops":1,"prepare.queue_time_us":526794,"prepare.run_cpu_time_us":625,"prepare.run_wall_time_us":623,"replication_time_us":93905}]]}
I20240503 02:12:30.196969 4273 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 678) took 1973 ms. Trace:
I20240503 02:12:30.197250 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6c80efe35343989f46a2c181f38129 (DEFAULT_TABLE table=default.Table_73a988b03f62487b86ace12f89bcd124 [id=65569d68a53e48d0ad75a523117d2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.197248 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c569a33635544c3189f27a82ffb3d32f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:30.197474 4330 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:29.827762 4499 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 497) took 1410 ms. Trace:
W20240503 02:12:30.197805 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c569a33635544c3189f27a82ffb3d32f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:30.197708 4499 rpcz_store.cc:276] 0503 02:12:28.417347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.773248 (+1355901us) service_pool.cc:225] Handling call
0503 02:12:29.773816 (+ 568us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.773841 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.805949 (+ 32108us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.827458 (+ 21509us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.827704 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.694800 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 (DEFAULT_TABLE table=default.Table_064299ed49e74e50a322f6cbf4878b98 [id=ed428cec915844f6bda0f7b5e9dd2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.792737 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:30.198180 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 (DEFAULT_TABLE table=default.Table_6a0d5b09dfb04a1892fb248b676a923b [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.198385 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:29.789353 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 79884b3e73904c708d0d48bca85351b7 (table default.Table_b839b2ab5a8a4a399e63762bec827155 [id=156d16ce2a4246cd81968e098a9227dc]) successfully deleted
I20240503 02:12:29.202517 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 39c48b14bc624c28a871d45e9429ee30 (DEFAULT_TABLE table=default.Table_b9f90c9e6ae3489b9809bed574c04644 [id=d5ef38cbaed14297b58cf74b9c85244e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:30.198616 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 409258us
I20240503 02:12:30.196873 4274 rpcz_store.cc:276] 0503 02:12:28.274960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.819184 (+544224us) service_pool.cc:225] Handling call
0503 02:12:28.991331 (+172147us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:28.991852 (+ 521us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:29.581638 (+589786us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:29.581789 (+ 151us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:29.581907 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.442893 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.442972 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.442976 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:29.442979 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:29.443043 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.443053 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.443058 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.443112 (+ 54us) tablet.cc:699] Row locks acquired
0503 02:12:29.443115 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:29.443140 (+ 25us) write_op.cc:270] Start()
0503 02:12:29.443162 (+ 22us) write_op.cc:276] Timestamp: P: 1714702349443108 usec, L: 0
0503 02:12:29.443165 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.443298 (+ 133us) log.cc:842] Serialized 433 byte log entry
0503 02:12:29.580270 (+136972us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.580858 (+ 588us) write_op.cc:301] APPLY: starting
0503 02:12:29.580888 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.580951 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.580985 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.581116 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.581152 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:29.581161 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:29.581249 (+ 88us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.581519 (+ 270us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.581538 (+ 19us) write_op.cc:454] Released schema lock
0503 02:12:29.581547 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.582148 (+ 601us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.582155 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":471,"apply.run_cpu_time_us":626,"apply.run_wall_time_us":1355,"num_ops":2,"prepare.queue_time_us":450474,"prepare.run_cpu_time_us":450,"prepare.run_wall_time_us":449,"replication_time_us":136878}]]}
I20240503 02:12:30.198998 5742 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:12:29.700055 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 211dfd4049684f5e8af927b2a19263b7 (DEFAULT_TABLE table=default.Table_00deed6cfb98406d84258657464b8f93 [id=8af4014e8da64c3aae761834a6683b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.700214 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 2848c837598c49628c11374d3c875b3e (DEFAULT_TABLE table=default.Table_a7c7733ae249477b9cbd82203830f7a6 [id=eede36b209e340bd872e8b845af5020c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.700446 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 6645c12ffce3474e9746249ce19a550a (DEFAULT_TABLE table=default.Table_2701a2118e9c4a24b91afbb68eebe26c [id=e1be5b08c5954211b146294a8b6e68f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.199340 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_896028d24de641e4b765e2a64b27186c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:30.199604 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 3478033c4ed94268a2436a2942c27a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:29.700762 4777 tablet_service.cc:1467] Processing CreateTablet for tablet f4012724bd7944fea411eb3c11b3feb7 (DEFAULT_TABLE table=default.Table_0bb92f02ee674d1f8317b3ee2e6cebb7 [id=2d4608c0683a419694b160e3be066cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.126161 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1a5d3ddc24b44b921648bd94ad827 (DEFAULT_TABLE table=default.Table_01fcdbe2fef14f7284c1f68c4f72e92c [id=29ccc9b87d064083ae19ae549e3db5a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:30.199858 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_896028d24de641e4b765e2a64b27186c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:29.789570 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet dedead5a2bf347f6991cce8e9b4db708 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.199991 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]) successfully deleted
I20240503 02:12:29.798408 4792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 516) took 1349 ms. Trace:
I20240503 02:12:30.200125 4792 rpcz_store.cc:276] 0503 02:12:28.449108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.449159 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:28.868516 (+419357us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:28.868546 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.780184 (+911638us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.798110 (+ 17926us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.798305 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.799276 4787 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 522) took 1260 ms. Trace:
I20240503 02:12:30.200491 4787 rpcz_store.cc:276] 0503 02:12:28.538578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.099710 (+561132us) service_pool.cc:225] Handling call
0503 02:12:29.785044 (+685334us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.785074 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.788626 (+ 3552us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.799081 (+ 10455us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.799226 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.809613 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e3d3a356f349d0960276be05346888. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:30.200641 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2a51563396614c54b980f0679e84a2b1 because the tablet was not found. No further retry: Not found: Tablet not found: 2a51563396614c54b980f0679e84a2b1
I20240503 02:12:29.202900 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 (DEFAULT_TABLE table=default.Table_e0f6872c37c746fab51a59df01834e38 [id=ad31fa5a307d4e12a48f892fe41a1d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.828822 4490 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 488) took 1493 ms. Trace:
I20240503 02:12:30.201037 4490 rpcz_store.cc:276] 0503 02:12:28.334904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.830007 (+495103us) service_pool.cc:225] Handling call
0503 02:12:29.076483 (+246476us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.076516 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.790454 (+713938us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.828596 (+ 38142us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.828768 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.244135 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:29.242080 4498 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 456) took 1707 ms. Trace:
I20240503 02:12:30.197088 4273 rpcz_store.cc:276] 0503 02:12:28.223268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.283197 (+ 59929us) service_pool.cc:225] Handling call
0503 02:12:28.345703 (+ 62506us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:28.827646 (+481943us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:28.827856 (+ 210us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:28.910506 (+ 82650us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:28.910675 (+ 169us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:29.530770 (+620095us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:30.196876 (+666106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.324965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.326328 (+ 1363us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.326457 (+ 129us) write_op.cc:435] Acquired schema lock
0503 02:12:29.326460 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:29.327400 (+ 940us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.327461 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.327468 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:29.328040 (+ 572us) tablet.cc:699] Row locks acquired
0503 02:12:29.328043 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:29.328329 (+ 286us) write_op.cc:270] Start()
0503 02:12:29.328686 (+ 357us) write_op.cc:276] Timestamp: P: 1714702349328298 usec, L: 0
0503 02:12:29.328811 (+ 125us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.330317 (+ 1506us) log.cc:842] Serialized 405 byte log entry
0503 02:12:29.527587 (+197270us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.529824 (+ 2237us) write_op.cc:301] APPLY: starting
0503 02:12:29.529843 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.529918 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.529991 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.530093 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.530157 (+ 64us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:29.530168 (+ 11us) write_op.cc:312] APPLY: finished
0503 02:12:29.530305 (+ 137us) log.cc:842] Serialized 23 byte log entry
0503 02:12:29.530625 (+ 320us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.530644 (+ 19us) write_op.cc:454] Released schema lock
0503 02:12:29.530652 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.530771 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.530778 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":137984384,"child_traces":[["op",{"apply.queue_time_us":2176,"apply.run_cpu_time_us":669,"apply.run_wall_time_us":1056,"num_ops":1,"prepare.queue_time_us":413266,"prepare.run_cpu_time_us":1699,"prepare.run_wall_time_us":6016,"replication_time_us":198759,"thread_start_us":7198,"threads_started":1}]]}
I20240503 02:12:30.201395 4498 rpcz_store.cc:276] 0503 02:12:27.534935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.535514 (+ 579us) service_pool.cc:225] Handling call
0503 02:12:27.535874 (+ 360us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:27.535897 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.910663 (+1374766us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.241865 (+331202us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.242033 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.201520 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 6e635f283288406aa16b3941be64e777 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.201773 4498 tablet_service.cc:1467] Processing CreateTablet for tablet f3a03a6dc04f47b29401aec463d0d442 (DEFAULT_TABLE table=default.Table_023bb9a5caa9417d9ab87c83b288de66 [id=33ee169586d84b0a95267f799469aeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.246372 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:30.008087 5668 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.202265 5668 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.202983 5668 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:29.246901 4646 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 482) took 1003 ms. Trace:
I20240503 02:12:30.203455 4646 rpcz_store.cc:276] 0503 02:12:28.243212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.243475 (+ 263us) service_pool.cc:225] Handling call
0503 02:12:28.835375 (+591900us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:28.835395 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:28.848232 (+ 12837us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.239258 (+391026us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.246835 (+ 7577us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":145255}
I20240503 02:12:29.090366 4608 leader_election.cc:302] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:29.259806 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 448) took 1685 ms. Trace:
I20240503 02:12:30.203958 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07225ae9ad3c4e44ac0c5c51421b0115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.204015 5115 rpcz_store.cc:276] 0503 02:12:27.573817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.573871 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:29.259530 (+1685659us) ts_tablet_manager.cc:1120] Deleting tablet d39f042afb954daa923502ef12377bd5
0503 02:12:29.259543 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:29.259728 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673770}
I20240503 02:12:30.204538 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 449) took 2622 ms. Trace:
I20240503 02:12:30.204624 5115 rpcz_store.cc:276] 0503 02:12:27.582045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.582604 (+ 559us) service_pool.cc:225] Handling call
0503 02:12:30.204351 (+2621747us) ts_tablet_manager.cc:1120] Deleting tablet 8241a88f5c3a47578c92343f9fb5f7cd
0503 02:12:30.204363 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:30.204484 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2617886}
I20240503 02:12:29.831506 4780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 523) took 1291 ms. Trace:
I20240503 02:12:29.283061 5106 tablet_bootstrap.cc:492] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:30.205106 5106 ts_tablet_manager.cc:1367] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 1.364s user 0.004s sys 0.008s
I20240503 02:12:30.205027 4780 rpcz_store.cc:276] 0503 02:12:28.539959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.099805 (+559846us) service_pool.cc:225] Handling call
0503 02:12:29.781335 (+681530us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.781363 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.784875 (+ 3512us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.831155 (+ 46280us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.831361 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:29.693632 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb (DEFAULT_TABLE table=default.Table_3c2d73cae23744ad8a1b256940d3a750 [id=430bf07595d347768337d56459901a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.003064 5791 raft_consensus.cc:2781] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:12:30.200879 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3b72c20a4b5a4d2ea266f031733d1aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b72c20a4b5a4d2ea266f031733d1aab already in progress: creating tablet
I20240503 02:12:30.205523 5791 raft_consensus.cc:482] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.205602 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fcfe6e2dc844a486476bf4be80d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.205675 5791 raft_consensus.cc:3037] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:30.205782 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d39f042afb954daa923502ef12377bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d39f042afb954daa923502ef12377bd5 already in progress: creating tablet
W20240503 02:12:30.206028 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 23ccd25074db40fb96053b915125a425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ccd25074db40fb96053b915125a425 already in progress: creating tablet
I20240503 02:12:30.205945 5106 raft_consensus.cc:348] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.206188 5106 raft_consensus.cc:374] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.206264 5106 raft_consensus.cc:729] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.206616 5106 consensus_queue.cc:260] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:29.564632 4590 kernel_stack_watchdog.cc:198] Thread 4608 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 162ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1c91292980 (unknown) at ??:0
@ 0x7f1c9128c9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f1c8b212904 google::LogMessage::Flush() at ??:0
@ 0x7f1c8b212a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1c996b83f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f1c996b9782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f1c996bc12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f1c996bbeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1c94b528cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f1c94b524ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1c94b3e4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f1c94b3ea2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f1c94adb51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f1c94ad9f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 02:12:30.046806 5793 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:30.220757 5793 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:29.772738 4670 rpcz_store.cc:276] 0503 02:12:27.847253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.847657 (+ 404us) service_pool.cc:225] Handling call
0503 02:12:27.849864 (+ 2207us) raft_consensus.cc:1457] Updating replica for 1 ops
0503 02:12:27.850023 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0503 02:12:27.850031 (+ 8us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0503 02:12:27.855777 (+ 5746us) alter_schema_op.cc:119] START. Timestamp: 1714702347838061
0503 02:12:28.842680 (+986903us) log.cc:842] Serialized 173 byte log entry
0503 02:12:28.961716 (+119036us) raft_consensus.cc:1631] Marking committed up to 4
0503 02:12:28.961732 (+ 16us) raft_consensus.cc:1683] Filling consensus response to leader.
0503 02:12:28.961776 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0503 02:12:29.099024 (+137248us) raft_consensus.cc:1671] finished
0503 02:12:29.099027 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0503 02:12:29.099089 (+ 62us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:28.257565 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:28.257684 (+ 119us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:28.257694 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:28.257700 (+ 6us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
Metrics: {"thread_start_us":118918,"threads_started":1,"wal-append.queue_time_us":254739,"child_traces":[["op",{"prepare.queue_time_us":401670,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":157,"thread_start_us":986522,"threads_started":1}]]}
I20240503 02:12:29.791873 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b0672199fc4d538242c38616f2fd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.196180 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20d0ca30f845618e0a20aa85dafbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.817526 4648 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 506) took 1307 ms. Trace:
I20240503 02:12:29.908314 5717 raft_consensus.cc:482] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.221853 5793 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:30.204886 5115 tablet_replica.cc:330] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:30.221944 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.213610 4503 rpcz_store.cc:276] 0503 02:12:28.335296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.074295 (+738999us) service_pool.cc:225] Handling call
0503 02:12:29.781100 (+706805us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.781142 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.792404 (+ 11262us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.817998 (+ 25594us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.819710 (+ 1712us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.086051 5796 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:30.222302 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e8fc6409b5448eaa567879c5b7d504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.222187 5796 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.222508 4503 tablet_service.cc:1467] Processing CreateTablet for tablet fe35f00dd4e74dc8b6b61eefe2bafaab (DEFAULT_TABLE table=default.Table_a2518b080992412aa7ae7abdd643bdf7 [id=fb33eccba1eb4943ae2bedcf0cd10e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.223019 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35f00dd4e74dc8b6b61eefe2bafaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.223026 5796 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:29.998836 4746 leader_election.cc:302] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:30.223318 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 225796us
I20240503 02:12:30.223650 4746 leader_election.cc:302] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:30.224277 4746 leader_election.cc:341] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:30.224591 4746 leader_election.cc:341] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:30.225032 5796 raft_consensus.cc:2781] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.198632 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:30.225139 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335f896171b54b67ad44cb8e161003ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.225152 5796 raft_consensus.cc:482] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.225244 5796 raft_consensus.cc:3037] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:30.228010 4267 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 88) took 1271 ms (client timeout 1000 ms). Trace:
W20240503 02:12:30.228183 4267 rpcz_store.cc:269] 0503 02:12:28.956694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.774630 (+817936us) service_pool.cc:225] Handling call
0503 02:12:30.227940 (+453310us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":13}
I20240503 02:12:30.228246 5796 raft_consensus.cc:504] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.228623 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 76b01907833d44f6a000a4de3a2a6e96 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:30.229080 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e08c10c15f41878202cef0c8d5ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.193828 4269 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 685) took 1696 ms. Trace:
I20240503 02:12:30.229323 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552dad16f30a410280c87a6c5ef0eaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.229650 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b8577a7774488aa768529960b5240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.229321 4269 rpcz_store.cc:276] 0503 02:12:28.497164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.830585 (+333421us) service_pool.cc:225] Handling call
0503 02:12:28.929773 (+ 99188us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:28.929965 (+ 192us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:28.930046 (+ 81us) catalog_manager.cc:2758] Locking tablets
0503 02:12:29.053395 (+123349us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:29.584128 (+530733us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:29.584243 (+ 115us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:29.584299 (+ 56us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:30.193758 (+609459us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.443385 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:29.443466 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:29.443470 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:29.443473 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:29.443561 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:29.443570 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:29.443574 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:29.443680 (+ 106us) tablet.cc:699] Row locks acquired
0503 02:12:29.443683 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:29.443711 (+ 28us) write_op.cc:270] Start()
0503 02:12:29.443734 (+ 23us) write_op.cc:276] Timestamp: P: 1714702349443678 usec, L: 0
0503 02:12:29.443739 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.443866 (+ 127us) log.cc:842] Serialized 725 byte log entry
0503 02:12:29.580823 (+136957us) op_driver.cc:444] REPLICATION: finished
0503 02:12:29.581289 (+ 466us) write_op.cc:301] APPLY: starting
0503 02:12:29.581365 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:29.581591 (+ 226us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:29.581817 (+ 226us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:29.582360 (+ 543us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:29.582596 (+ 236us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:29.582670 (+ 74us) write_op.cc:312] APPLY: finished
0503 02:12:29.583199 (+ 529us) log.cc:842] Serialized 29 byte log entry
0503 02:12:29.583745 (+ 546us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.583829 (+ 84us) write_op.cc:454] Released schema lock
0503 02:12:29.583899 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:29.584283 (+ 384us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:29.584431 (+ 148us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":1893,"apply.run_wall_time_us":6024,"num_ops":2,"prepare.queue_time_us":389188,"prepare.run_cpu_time_us":536,"prepare.run_wall_time_us":535,"replication_time_us":136927}]]}
I20240503 02:12:30.229856 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e635f283288406aa16b3941be64e777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.230078 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a03a6dc04f47b29401aec463d0d442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.230108 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7ea27632721644359e27e0917381fb1c" } modify_external_catalogs: true
I20240503 02:12:30.230286 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7ea27632721644359e27e0917381fb1c" } modify_external_catalogs: true
I20240503 02:12:30.230792 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7900af2ce5034eea9bccdbdf3ea5b709. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:30.224866 4746 leader_election.cc:341] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:30.232586 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6c80efe35343989f46a2c181f38129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.235013 5103 tablet_bootstrap.cc:654] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.235056 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f6279f16dc42d69e794e3c6586163d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.236488 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8135907b78284d7ca74396cc77dd0612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.237458 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72bff3c50eb4ac6b1b2feb7250ba43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.237720 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c43838340e44ef92e318a969e34af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.208667 5791 raft_consensus.cc:504] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.213382 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4012724bd7944fea411eb3c11b3feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.229161 5793 raft_consensus.cc:2781] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.249058 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.249363 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848c837598c49628c11374d3c875b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.242671 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.244267 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.245468 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.318881 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4312
I20240503 02:12:30.269783 4267 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 692) took 1262 ms. Trace:
I20240503 02:12:30.943305 4267 rpcz_store.cc:276] 0503 02:12:29.007022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.228533 (+1221511us) service_pool.cc:225] Handling call
0503 02:12:30.269708 (+ 41175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.946605 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ea8c1983dad340faaa278bfe0d6ea103" } new_table_name: "default.Table_72f299109c02426da8ff1ae9d7f1a212" modify_external_catalogs: true
I20240503 02:12:30.259819 5115 raft_consensus.cc:2229] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:30.948295 5115 raft_consensus.cc:2258] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:30.259749 5717 raft_consensus.cc:504] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.259783 4648 rpcz_store.cc:276] 0503 02:12:28.509429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.031857 (+522428us) service_pool.cc:225] Handling call
0503 02:12:29.044522 (+ 12665us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.044550 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:29.802798 (+758248us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:29.814617 (+ 11819us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:29.816830 (+ 2213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.949108 5115 ts_tablet_manager.cc:1875] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:30.228886 5796 leader_election.cc:288] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.259749 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1a5d3ddc24b44b921648bd94ad827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.249089 4508 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 490) took 1913 ms. Trace:
I20240503 02:12:30.949445 4508 rpcz_store.cc:276] 0503 02:12:28.335066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.074295 (+739229us) service_pool.cc:225] Handling call
0503 02:12:29.771979 (+697684us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.772000 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.225911 (+453911us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.228810 (+ 2899us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.248939 (+ 20129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.949753 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35f00dd4e74dc8b6b61eefe2bafaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.950162 5717 leader_election.cc:288] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.950125 4508 tablet_service.cc:1467] Processing CreateTablet for tablet 22b0672199fc4d538242c38616f2fd80 (DEFAULT_TABLE table=default.Table_19cd1048ca9d40cb9bfd8f1267a99e0c [id=2c71121bb96e44ffbe0f8d99dbc30d17]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:30.272168 4271 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.66:56851 (request call id 72) took 1309 ms (client timeout 1000 ms). Trace:
W20240503 02:12:30.950582 4271 rpcz_store.cc:269] 0503 02:12:28.962465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.954793 (+992328us) service_pool.cc:225] Handling call
0503 02:12:30.272098 (+317305us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240503 02:12:30.951064 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fb11bd168d9b4a22940b384400a5c6d5" } modify_external_catalogs: true
W20240503 02:12:30.339298 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 1.23342e+06us
I20240503 02:12:30.951294 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fb11bd168d9b4a22940b384400a5c6d5" } modify_external_catalogs: true
I20240503 02:12:30.287993 4491 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 504) took 1852 ms. Trace:
I20240503 02:12:30.295449 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1f8b305ac6424a8bc6a9cf25eec587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.953027 4491 rpcz_store.cc:276] 0503 02:12:28.426259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.196876 (+1770617us) service_pool.cc:225] Handling call
0503 02:12:30.197469 (+ 593us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.197506 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.236140 (+ 38634us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.278866 (+ 42726us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.279080 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.953301 4376 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:30.246929 5791 leader_election.cc:288] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.953616 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 39c48b14bc624c28a871d45e9429ee30 (DEFAULT_TABLE table=default.Table_b9f90c9e6ae3489b9809bed574c04644 [id=d5ef38cbaed14297b58cf74b9c85244e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.954079 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c48b14bc624c28a871d45e9429ee30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.272243 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:30.420547 4268 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 688) took 1923 ms. Trace:
I20240503 02:12:30.468441 4272 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 689) took 1970 ms. Trace:
I20240503 02:12:30.473918 4270 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 690) took 1976 ms. Trace:
I20240503 02:12:30.338786 5803 raft_consensus.cc:2781] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.956295 5803 raft_consensus.cc:482] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.956454 5803 raft_consensus.cc:3037] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.956187 4270 rpcz_store.cc:276] 0503 02:12:28.497578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.196976 (+1699398us) service_pool.cc:225] Handling call
0503 02:12:30.197895 (+ 919us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:30.198398 (+ 503us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:30.473616 (+275218us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:30.473765 (+ 149us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:30.473869 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.464061 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.464316 (+ 255us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.464323 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:30.464325 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:30.464526 (+ 201us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.465111 (+ 585us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.465119 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.465259 (+ 140us) tablet.cc:699] Row locks acquired
0503 02:12:30.465264 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:30.468600 (+ 3336us) write_op.cc:270] Start()
0503 02:12:30.468668 (+ 68us) write_op.cc:276] Timestamp: P: 1714702350465301 usec, L: 0
0503 02:12:30.468674 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.469106 (+ 432us) log.cc:842] Serialized 433 byte log entry
0503 02:12:30.472886 (+ 3780us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.473021 (+ 135us) write_op.cc:301] APPLY: starting
0503 02:12:30.473040 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.473090 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.473132 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.473259 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.473308 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:30.473318 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:30.473413 (+ 95us) log.cc:842] Serialized 29 byte log entry
0503 02:12:30.473481 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.473504 (+ 23us) write_op.cc:454] Released schema lock
0503 02:12:30.473513 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.473576 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.473580 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":606,"apply.run_wall_time_us":605,"num_ops":2,"prepare.queue_time_us":265139,"prepare.run_cpu_time_us":1143,"prepare.run_wall_time_us":6099,"raft.queue_time_us":56,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":768,"replication_time_us":4185}]]}
I20240503 02:12:30.957347 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d54df163e4634a779d3bcfddb083db60" } modify_external_catalogs: true
I20240503 02:12:30.957494 4643 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 515) took 2419 ms. Trace:
I20240503 02:12:30.957577 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d54df163e4634a779d3bcfddb083db60" } modify_external_catalogs: true
I20240503 02:12:30.957630 4643 rpcz_store.cc:276] 0503 02:12:28.537607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.149173 (+611566us) service_pool.cc:225] Handling call
0503 02:12:29.969311 (+820138us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.969337 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.953824 (+984487us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.957262 (+ 3438us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.957441 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22144}
W20240503 02:12:30.958084 4266 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 64) took 1864 ms (client timeout 1000 ms). Trace:
W20240503 02:12:30.958223 4266 rpcz_store.cc:269] 0503 02:12:29.093307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.955319 (+862012us) service_pool.cc:225] Handling call
0503 02:12:30.958018 (+1002699us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6}
I20240503 02:12:30.958652 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8af9d8455d574ede92363cd1cafb1334" } new_table_name: "default.Table_9947777ee256419394d206f28a1a6e11" modify_external_catalogs: true
I20240503 02:12:30.482940 4274 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 691) took 1648 ms. Trace:
I20240503 02:12:30.302326 4495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 496) took 1871 ms. Trace:
I20240503 02:12:30.334759 4509 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 492) took 1999 ms. Trace:
I20240503 02:12:30.343192 4499 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 505) took 1911 ms. Trace:
I20240503 02:12:30.344610 4502 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 483) took 2039 ms. Trace:
I20240503 02:12:30.345024 4497 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 500) took 1926 ms. Trace:
I20240503 02:12:30.960286 4497 rpcz_store.cc:276] 0503 02:12:28.418384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.954885 (+1536501us) service_pool.cc:225] Handling call
0503 02:12:29.955284 (+ 399us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.955310 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.267306 (+311996us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.344806 (+ 77500us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.344980 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.959959 4274 rpcz_store.cc:276] 0503 02:12:28.834321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.199075 (+1364754us) service_pool.cc:225] Handling call
0503 02:12:30.199950 (+ 875us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:30.200411 (+ 461us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:30.482591 (+282180us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:30.482762 (+ 171us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:30.482875 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.470185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.470381 (+ 196us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.470388 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:30.470426 (+ 38us) tablet.cc:660] Decoding operations
0503 02:12:30.470567 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.470577 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.470582 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.471226 (+ 644us) tablet.cc:699] Row locks acquired
0503 02:12:30.471231 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:30.474217 (+ 2986us) write_op.cc:270] Start()
0503 02:12:30.474282 (+ 65us) write_op.cc:276] Timestamp: P: 1714702350471267 usec, L: 0
0503 02:12:30.474289 (+ 7us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.474612 (+ 323us) log.cc:842] Serialized 433 byte log entry
0503 02:12:30.481519 (+ 6907us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.482048 (+ 529us) write_op.cc:301] APPLY: starting
0503 02:12:30.482066 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.482115 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.482142 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.482269 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.482311 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:30.482318 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:30.482399 (+ 81us) log.cc:842] Serialized 29 byte log entry
0503 02:12:30.482470 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.482493 (+ 23us) write_op.cc:454] Released schema lock
0503 02:12:30.482503 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.482570 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.482576 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":452,"apply.run_cpu_time_us":586,"apply.run_wall_time_us":585,"mutex_wait_us":1751,"num_ops":2,"prepare.queue_time_us":269246,"prepare.run_cpu_time_us":1003,"prepare.run_wall_time_us":6319,"raft.queue_time_us":3312,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":877,"replication_time_us":7182}]]}
I20240503 02:12:30.960650 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet fdc2850c07d042a7951a5fbeb0e4945d from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:30.372067 5106 ts_tablet_manager.cc:1398] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.167s user 0.004s sys 0.005s
I20240503 02:12:30.960950 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 (DEFAULT_TABLE table=default.Table_e0f6872c37c746fab51a59df01834e38 [id=ad31fa5a307d4e12a48f892fe41a1d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.961441 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.253206 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1a5d3ddc24b44b921648bd94ad827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.255101 4793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 512) took 1817 ms. Trace:
I20240503 02:12:30.484660 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5b91a3f3cad74f6aa6035a013cca651f
I20240503 02:12:30.961743 4793 rpcz_store.cc:276] 0503 02:12:28.437198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.437247 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:28.868746 (+431499us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:28.868769 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.207760 (+1338991us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.253300 (+ 45540us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.254937 (+ 1637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.305867 5793 raft_consensus.cc:686] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:30.962177 4269 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 693) took 1926 ms. Trace:
I20240503 02:12:30.962227 5793 consensus_queue.cc:237] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.827869 5804 raft_consensus.cc:482] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:30.315064 4778 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 519) took 1773 ms. Trace:
I20240503 02:12:30.963384 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08e7d17f1644749a9a2e7175b3191a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:30.358278 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 65340.7us
I20240503 02:12:30.963510 4778 rpcz_store.cc:276] 0503 02:12:28.529349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.067613 (+538264us) service_pool.cc:225] Handling call
0503 02:12:29.087933 (+ 20320us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.088011 (+ 78us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.204003 (+1115992us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.302316 (+ 98313us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.303206 (+ 890us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:30.964943 4746 connection.cc:232] RPC call timeout handler was delayed by -1.9228703180000366s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
I20240503 02:12:30.962306 4269 rpcz_store.cc:276] 0503 02:12:29.035863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.229886 (+1194023us) service_pool.cc:225] Handling call
0503 02:12:30.230411 (+ 525us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:30.270179 (+ 39768us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:30.270264 (+ 85us) catalog_manager.cc:2758] Locking tablets
0503 02:12:30.270486 (+ 222us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:30.484228 (+213742us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:30.484342 (+ 114us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:30.484395 (+ 53us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:30.962138 (+477743us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.476547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.476673 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.476678 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:30.476681 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:30.476817 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.476828 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.476834 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.477028 (+ 194us) tablet.cc:699] Row locks acquired
0503 02:12:30.477033 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:30.477072 (+ 39us) write_op.cc:270] Start()
0503 02:12:30.477104 (+ 32us) write_op.cc:276] Timestamp: P: 1714702350477042 usec, L: 0
0503 02:12:30.477110 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.477328 (+ 218us) log.cc:842] Serialized 725 byte log entry
0503 02:12:30.482971 (+ 5643us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.483313 (+ 342us) write_op.cc:301] APPLY: starting
0503 02:12:30.483326 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.483359 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.483397 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.483455 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.483481 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:30.483487 (+ 6us) write_op.cc:312] APPLY: finished
0503 02:12:30.483539 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 02:12:30.483592 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.483607 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:30.483612 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.483655 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.483658 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":152,"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":388,"apply.run_wall_time_us":401,"num_ops":2,"prepare.queue_time_us":205224,"prepare.run_cpu_time_us":996,"prepare.run_wall_time_us":1504,"raft.queue_time_us":778,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":15770,"replication_time_us":5823}]]}
W20240503 02:12:30.555137 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_df82f492a3db4e1480369126b97c1169 [id=eff7b93c47234c2fa5a3d6ecb95c5cae], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 17c2a8190210439fb7589cb02ec82bc2 consensus state has no leader
I20240503 02:12:30.968536 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:30.956121 4268 rpcz_store.cc:276] 0503 02:12:28.497433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.189800 (+1692367us) service_pool.cc:225] Handling call
0503 02:12:30.191112 (+ 1312us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:30.191774 (+ 662us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:30.420135 (+228361us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:30.420338 (+ 203us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:30.420470 (+ 132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.345930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.352530 (+ 6600us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.352768 (+ 238us) write_op.cc:435] Acquired schema lock
0503 02:12:30.352774 (+ 6us) tablet.cc:660] Decoding operations
0503 02:12:30.376388 (+ 23614us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.377787 (+ 1399us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.378034 (+ 247us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.382114 (+ 4080us) tablet.cc:699] Row locks acquired
0503 02:12:30.382345 (+ 231us) write_op.cc:260] PREPARE: finished
0503 02:12:30.384905 (+ 2560us) write_op.cc:270] Start()
0503 02:12:30.386855 (+ 1950us) write_op.cc:276] Timestamp: P: 1714702350384016 usec, L: 0
0503 02:12:30.386867 (+ 12us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.388302 (+ 1435us) log.cc:842] Serialized 433 byte log entry
0503 02:12:30.397622 (+ 9320us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.411995 (+ 14373us) write_op.cc:301] APPLY: starting
0503 02:12:30.412508 (+ 513us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.413501 (+ 993us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.414154 (+ 653us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.416024 (+ 1870us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.416574 (+ 550us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:30.416833 (+ 259us) write_op.cc:312] APPLY: finished
0503 02:12:30.418280 (+ 1447us) log.cc:842] Serialized 29 byte log entry
0503 02:12:30.419190 (+ 910us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.419403 (+ 213us) write_op.cc:454] Released schema lock
0503 02:12:30.419414 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.420990 (+ 1576us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.421245 (+ 255us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13774,"apply.run_cpu_time_us":1501,"apply.run_wall_time_us":10097,"num_ops":2,"prepare.queue_time_us":149551,"prepare.run_cpu_time_us":37427,"prepare.run_wall_time_us":117334,"raft.queue_time_us":615,"raft.run_cpu_time_us":824,"raft.run_wall_time_us":849,"replication_time_us":10725,"spinlock_wait_cycles":363904,"thread_start_us":232054,"threads_started":3}]]}
I20240503 02:12:31.501847 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:31.502175 4275 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:30.956151 4272 rpcz_store.cc:276] 0503 02:12:28.497505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.193375 (+1695870us) service_pool.cc:225] Handling call
0503 02:12:30.194393 (+ 1018us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:30.194913 (+ 520us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:30.468081 (+273168us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:30.468235 (+ 154us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:30.468364 (+ 129us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.460823 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.461118 (+ 295us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.461153 (+ 35us) write_op.cc:435] Acquired schema lock
0503 02:12:30.461156 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:30.461374 (+ 218us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.461424 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.461433 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.461667 (+ 234us) tablet.cc:699] Row locks acquired
0503 02:12:30.461673 (+ 6us) write_op.cc:260] PREPARE: finished
0503 02:12:30.461748 (+ 75us) write_op.cc:270] Start()
0503 02:12:30.461824 (+ 76us) write_op.cc:276] Timestamp: P: 1714702350461691 usec, L: 0
0503 02:12:30.461829 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.462270 (+ 441us) log.cc:842] Serialized 433 byte log entry
0503 02:12:30.467115 (+ 4845us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.467250 (+ 135us) write_op.cc:301] APPLY: starting
0503 02:12:30.467272 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.467328 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.467368 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.467541 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.467589 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:30.467600 (+ 11us) write_op.cc:312] APPLY: finished
0503 02:12:30.467694 (+ 94us) log.cc:842] Serialized 29 byte log entry
0503 02:12:30.467764 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.467789 (+ 25us) write_op.cc:454] Released schema lock
0503 02:12:30.467798 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.467865 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.467873 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":679,"apply.run_wall_time_us":679,"num_ops":2,"prepare.queue_time_us":265312,"prepare.run_cpu_time_us":1369,"prepare.run_wall_time_us":3172,"raft.queue_time_us":265,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":1133,"replication_time_us":5192,"spinlock_wait_cycles":132992}]]}
I20240503 02:12:31.503409 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_54ab74ddb0d94515895951df323ebd48" } new_table_name: "default.Table_2aec4967b23a414596b8b354cf8de2d9" modify_external_catalogs: true
I20240503 02:12:31.365851 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4312
I20240503 02:12:30.875648 5829 raft_consensus.cc:482] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:31.504354 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:31.504479 5829 raft_consensus.cc:504] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:30.961091 5106 ts_tablet_manager.cc:1430] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Tablet startup took 755ms
I20240503 02:12:31.505623 5829 leader_election.cc:288] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:30.965570 4509 rpcz_store.cc:276] 0503 02:12:28.335413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.074812 (+739399us) service_pool.cc:225] Handling call
0503 02:12:29.954697 (+879885us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.954718 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.241925 (+287207us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.334493 (+ 92568us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.334687 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:31.505530 5106 ts_tablet_manager.cc:1432] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Trace:
0503 02:12:20.174964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.175045 (+ 81us) service_pool.cc:225] Handling call
0503 02:12:20.175545 (+ 500us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.175577 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.179121 (+ 3544us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.196651 (+ 17530us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.196787 (+ 136us) inbound_call.cc:173] Queueing success response
0503 02:12:28.840685 (+8643898us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:30.205203 (+1364518us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:30.205391 (+ 188us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:30.205570 (+ 179us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":154613,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":217,"tablet-open.queue_time_us":8643923,"thread_start_us":164803,"threads_started":1}
I20240503 02:12:31.504895 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:31.505034 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
I20240503 02:12:31.506317 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 7099b64f41e54cd7b7967c52892b2200 (DEFAULT_TABLE table=default.Table_923e68dde96e47489191170eb93b19ab [id=f2685456ff4046da8abb2ce85b4a8cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.506443 5106 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:31.506532 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:30.359107 4784 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 521) took 1811 ms. Trace:
I20240503 02:12:30.962980 5804 raft_consensus.cc:504] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.506814 4784 rpcz_store.cc:276] 0503 02:12:28.534983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.099546 (+564563us) service_pool.cc:225] Handling call
0503 02:12:29.960072 (+860526us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.960097 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.305441 (+345344us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.346234 (+ 40793us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.346404 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:30.981736 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 584694f1cc284a718f0e5fa15785d649 consensus state has no leader
I20240503 02:12:31.507203 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a90ebcf29004d1a925b69fa20998dd3 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:31.507387 4267 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 694) took 2471 ms. Trace:
I20240503 02:12:31.507365 5804 leader_election.cc:288] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.963150 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e3d3a356f349d0960276be05346888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.507679 4268 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 700) took 2316 ms. Trace:
I20240503 02:12:31.507936 4268 rpcz_store.cc:276] 0503 02:12:29.191712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.507119 (+2315407us) service_pool.cc:225] Handling call
0503 02:12:31.507610 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.508599 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet ec641e403ed843fda31dabd3920aeb68 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:30.346792 4506 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 489) took 2011 ms. Trace:
I20240503 02:12:31.509155 4506 rpcz_store.cc:276] 0503 02:12:28.334985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.849468 (+514483us) service_pool.cc:225] Handling call
0503 02:12:29.076395 (+226927us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.076419 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.288025 (+1211606us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.346601 (+ 58576us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.346741 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":127}
W20240503 02:12:31.509560 4273 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 65) took 1310 ms (client timeout 1000 ms). Trace:
W20240503 02:12:31.509691 4273 rpcz_store.cc:269] 0503 02:12:30.199546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.201411 (+ 1865us) service_pool.cc:225] Handling call
0503 02:12:31.509516 (+1308105us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11}
I20240503 02:12:31.509804 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 44707b6439b141b7b63f49f34f25b37a (DEFAULT_TABLE table=default.Table_9b6b0e4d8d9f43bf9322382e80b8e842 [id=74f5d88b2f9d4bbe941ebc9fddc18e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.510051 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet fcfb174825f1441f861ae6816137f855 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:30.372642 5105 ts_tablet_manager.cc:1398] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.179s user 0.006s sys 0.006s
W20240503 02:12:31.511062 5105 ts_tablet_manager.cc:1430] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Tablet startup took 1317ms
W20240503 02:12:31.511158 5105 ts_tablet_manager.cc:1432] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Trace:
0503 02:12:20.427222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.427285 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:20.427745 (+ 460us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.427769 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.431289 (+ 3520us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.451542 (+ 20253us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.451686 (+ 144us) inbound_call.cc:173] Queueing success response
0503 02:12:28.830001 (+8378315us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:30.193425 (+1363424us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:30.193617 (+ 192us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:30.193812 (+ 195us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":26860,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"tablet-open.queue_time_us":8378342,"thread_start_us":176998,"threads_started":1}
I20240503 02:12:31.512115 5105 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:31.003347 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d39f042afb954daa923502ef12377bd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d39f042afb954daa923502ef12377bd5 already in progress: creating tablet
W20240503 02:12:31.513339 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 510041us
W20240503 02:12:31.513662 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8241a88f5c3a47578c92343f9fb5f7cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8241a88f5c3a47578c92343f9fb5f7cd already in progress: creating tablet
I20240503 02:12:31.514907 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.971608 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:30.348079 4492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 485) took 2014 ms. Trace:
I20240503 02:12:31.515820 4492 rpcz_store.cc:276] 0503 02:12:28.333843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.404305 (+ 70462us) service_pool.cc:225] Handling call
0503 02:12:29.075898 (+671593us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.075951 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.254528 (+1178577us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.347897 (+ 93369us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.348023 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.516430 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 030dc0d3eeb445da91cfeef6886b651d (DEFAULT_TABLE table=default.Table_5c4057da7da24b3e8c9e3e1a4875666e [id=005c67d81b5445deb1e31f613797233c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:31.238713 4248 kernel_stack_watchdog.cc:198] Thread 4259 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 114ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fd3ea573980 (unknown) at ??:0
@ 0x7fd3ea56d9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fd3e44f3904 google::LogMessage::Flush() at ??:0
@ 0x7fd3e44f3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd3fa695465 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7fd3fa5ecb09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fd3fa697e30 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fd3fa697c4e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fd3ede338cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fd3ede334ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fd3ede1f4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fd3ede1fa2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fd3eddbc51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fd3eddbaf45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 02:12:31.382885 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 48e713ee37d042309166b0f118e14986
I20240503 02:12:30.486809 5808 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:31.517068 5808 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.971747 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:31.517966 5808 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.356987 4500 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 498) took 1938 ms. Trace:
I20240503 02:12:31.391041 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:31.518247 4500 rpcz_store.cc:276] 0503 02:12:28.418140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.953797 (+1535657us) service_pool.cc:225] Handling call
0503 02:12:29.954409 (+ 612us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.954442 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.299265 (+344823us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.355607 (+ 56342us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.356941 (+ 1334us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29}
I20240503 02:12:31.518868 4270 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 696) took 2482 ms. Trace:
I20240503 02:12:31.518847 4500 tablet_service.cc:1467] Processing CreateTablet for tablet b0e3d3a356f349d0960276be05346888 (DEFAULT_TABLE table=default.Table_ff6cfc8e7a2d4b8a8f691caa77265f87 [id=08935b6cc24d41af8359f67794027b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.971882 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:31.519248 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60837d0810041cc883d3d8056d11bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.518973 4270 rpcz_store.cc:276] 0503 02:12:29.036218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.957121 (+1920903us) service_pool.cc:225] Handling call
0503 02:12:30.957691 (+ 570us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:30.957844 (+ 153us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:30.957914 (+ 70us) catalog_manager.cc:2758] Locking tablets
0503 02:12:30.958071 (+ 157us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:31.390810 (+432739us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:31.390913 (+ 103us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:31.390974 (+ 61us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:31.518807 (+127833us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.986353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.986516 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.986524 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:30.986526 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:30.986658 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.986671 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.986676 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.986793 (+ 117us) tablet.cc:699] Row locks acquired
0503 02:12:30.986795 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:30.986834 (+ 39us) write_op.cc:270] Start()
0503 02:12:30.986868 (+ 34us) write_op.cc:276] Timestamp: P: 1714702350986787 usec, L: 0
0503 02:12:30.986872 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.987038 (+ 166us) log.cc:842] Serialized 756 byte log entry
0503 02:12:31.389925 (+402887us) op_driver.cc:444] REPLICATION: finished
0503 02:12:31.390251 (+ 326us) write_op.cc:301] APPLY: starting
0503 02:12:31.390269 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:31.390313 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:31.390343 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:31.390426 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:31.390462 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:31.390470 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:31.390567 (+ 97us) log.cc:842] Serialized 29 byte log entry
0503 02:12:31.390643 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.390666 (+ 23us) write_op.cc:454] Released schema lock
0503 02:12:31.390675 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:31.390742 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.390746 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":27435,"prepare.run_cpu_time_us":738,"prepare.run_wall_time_us":738,"replication_time_us":403022}]]}
I20240503 02:12:30.359970 4504 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 482) took 2121 ms. Trace:
I20240503 02:12:31.519609 4504 rpcz_store.cc:276] 0503 02:12:28.284233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.284374 (+ 141us) service_pool.cc:225] Handling call
0503 02:12:29.076320 (+791946us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.076342 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.303530 (+1227188us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.359716 (+ 56186us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.359901 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328}
I20240503 02:12:31.519878 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3d11cc9d7e0242eeb996280bfa1f9e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:31.520119 4504 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1a5d3ddc24b44b921648bd94ad827 (DEFAULT_TABLE table=default.Table_01fcdbe2fef14f7284c1f68c4f72e92c [id=29ccc9b87d064083ae19ae549e3db5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.972010 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:31.520452 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d11cc9d7e0242eeb996280bfa1f9e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:30.513551 5801 raft_consensus.cc:482] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:30.360124 4503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 508) took 1917 ms. Trace:
I20240503 02:12:31.520772 4503 rpcz_store.cc:276] 0503 02:12:28.438120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.222176 (+1784056us) service_pool.cc:225] Handling call
0503 02:12:30.222733 (+ 557us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.222769 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.316848 (+ 94079us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.355612 (+ 38764us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.355772 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.521239 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 (DEFAULT_TABLE table=default.Table_b3dcd45817264953aa80592e5bd4c4e4 [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.959439 5803 raft_consensus.cc:504] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.522176 5803 leader_election.cc:288] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.362069 4490 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 506) took 1927 ms. Trace:
W20240503 02:12:31.391497 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cd296bb3b574d6cbb0252ad10ada00c consensus state has no leader
I20240503 02:12:31.522384 4490 rpcz_store.cc:276] 0503 02:12:28.434520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.201261 (+1766741us) service_pool.cc:225] Handling call
0503 02:12:30.201673 (+ 412us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.201694 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.289979 (+ 88285us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.361838 (+ 71859us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.362007 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.768908 4791 tablet_service.cc:1467] Processing CreateTablet for tablet d31e874e285c4b20b4547048a3150d4a (DEFAULT_TABLE table=default.Table_896028d24de641e4b765e2a64b27186c [id=c1434f171fe84c7fa154776740a7b6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.367820 4494 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 502) took 1946 ms. Trace:
I20240503 02:12:30.373150 4507 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 495) took 1941 ms. Trace:
I20240503 02:12:31.522830 4490 tablet_service.cc:1467] Processing CreateTablet for tablet d40ed707ad074e0fbb0ee48c2137c661 (DEFAULT_TABLE table=default.Table_081e142b6f9e45b5b4cb3987b8f31ce1 [id=3829412f782f4f8ca43a8d851c0ec4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.382436 4501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 503) took 1961 ms. Trace:
I20240503 02:12:31.522958 4494 rpcz_store.cc:276] 0503 02:12:28.421063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.963993 (+1542930us) service_pool.cc:225] Handling call
0503 02:12:29.964467 (+ 474us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.964494 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.316068 (+351574us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.367580 (+ 51512us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.367770 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.523030 4501 rpcz_store.cc:276] 0503 02:12:28.421230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.192583 (+1771353us) service_pool.cc:225] Handling call
0503 02:12:30.193170 (+ 587us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.193207 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.276064 (+ 82857us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.381633 (+105569us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.382278 (+ 645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.982990 5717 consensus_queue.cc:1035] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 02:12:31.523331 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:31.523433 4494 tablet_service.cc:1467] Processing CreateTablet for tablet b60837d0810041cc883d3d8056d11bb5 (DEFAULT_TABLE table=default.Table_e57bbe28945b49eb9a52ef6f4170ca13 [id=669f20def97e41449baf70a130d948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.410490 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fdc2850c07d042a7951a5fbeb0e4945d
I20240503 02:12:30.776407 4777 tablet_service.cc:1467] Processing CreateTablet for tablet f2c283c08fe3453899834ff337798fc2 (DEFAULT_TABLE table=default.Table_b12b7f4b0c7048f38687ebc605d55be3 [id=ff255d3f86d64fb89058519936513875]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.784049 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 0700d4aed530413983381d1dc4f78ef9 (DEFAULT_TABLE table=default.Table_1e4923fbee8e4eb59344e61bfd2b499f [id=565babc2a39249b2bff0b5df1461a46b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.383682 4496 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 501) took 1961 ms. Trace:
I20240503 02:12:31.524027 4274 catalog_manager.cc:6554] ReplaceTablet: tablet fdc2850c07d042a7951a5fbeb0e4945d deleted and replaced by tablet b2560f50415444b8b2e497453a558d3a
I20240503 02:12:31.524078 4496 rpcz_store.cc:276] 0503 02:12:28.420942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.956358 (+1535416us) service_pool.cc:225] Handling call
0503 02:12:29.956714 (+ 356us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.956733 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.292735 (+336002us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.375242 (+ 82507us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.382581 (+ 7339us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.524168 4274 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 698) took 2414 ms. Trace:
I20240503 02:12:31.524305 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07225ae9ad3c4e44ac0c5c51421b0115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.524371 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 4064ff59de474e0db3a578eae10c3fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:31.524566 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:31.524282 4274 rpcz_store.cc:276] 0503 02:12:29.110049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.960544 (+1850495us) service_pool.cc:225] Handling call
0503 02:12:31.524131 (+563587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.987628 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.987697 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.987701 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:30.987703 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:30.987774 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.987782 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.987785 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.987834 (+ 49us) tablet.cc:699] Row locks acquired
0503 02:12:30.987836 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:30.987857 (+ 21us) write_op.cc:270] Start()
0503 02:12:30.987876 (+ 19us) write_op.cc:276] Timestamp: P: 1714702350987815 usec, L: 0
0503 02:12:30.987879 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.988002 (+ 123us) log.cc:842] Serialized 622 byte log entry
0503 02:12:31.402586 (+414584us) op_driver.cc:444] REPLICATION: finished
0503 02:12:31.403730 (+ 1144us) write_op.cc:301] APPLY: starting
0503 02:12:31.403873 (+ 143us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:31.404419 (+ 546us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:31.405492 (+ 1073us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:31.407143 (+ 1651us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:31.407678 (+ 535us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:31.407753 (+ 75us) write_op.cc:312] APPLY: finished
0503 02:12:31.408456 (+ 703us) log.cc:842] Serialized 29 byte log entry
0503 02:12:31.409625 (+ 1169us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.409857 (+ 232us) write_op.cc:454] Released schema lock
0503 02:12:31.409935 (+ 78us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:31.411373 (+ 1438us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.411446 (+ 73us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":703,"apply.run_cpu_time_us":3135,"apply.run_wall_time_us":8321,"num_ops":2,"prepare.queue_time_us":5444,"prepare.run_cpu_time_us":411,"prepare.run_wall_time_us":410,"replication_time_us":414621}]]}
I20240503 02:12:30.788856 4792 tablet_service.cc:1467] Processing CreateTablet for tablet f20e573d70bd46aab33466ccfc801465 (DEFAULT_TABLE table=default.Table_c569a33635544c3189f27a82ffb3d32f [id=a22896758eab4445a7df80811eca9dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.525097 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7fe2fd4286f4411aad562211e4bd77ae" } modify_external_catalogs: true
I20240503 02:12:30.971922 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9327a9e5472b4eda854320c3a95f8e56" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:30.420859 5103 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:31.525535 5103 ts_tablet_manager.cc:1367] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 2.575s user 0.022s sys 0.074s
I20240503 02:12:30.971794 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:30.545311 4498 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 507) took 2105 ms. Trace:
I20240503 02:12:31.526206 4498 rpcz_store.cc:276] 0503 02:12:28.436370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.201589 (+1765219us) service_pool.cc:225] Handling call
0503 02:12:30.201918 (+ 329us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.201939 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.296405 (+ 94466us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.499400 (+202995us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.541063 (+ 41663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.526430 5103 raft_consensus.cc:348] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.526675 5103 raft_consensus.cc:374] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.070405 5841 raft_consensus.cc:482] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.526777 5103 raft_consensus.cc:729] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.526926 5841 raft_consensus.cc:504] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.527061 5103 consensus_queue.cc:260] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.522958 4507 rpcz_store.cc:276] 0503 02:12:28.417000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.203735 (+786735us) service_pool.cc:225] Handling call
0503 02:12:29.956875 (+753140us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.956891 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.228432 (+271541us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.348803 (+120371us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.355780 (+ 6977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.527696 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44707b6439b141b7b63f49f34f25b37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.878365 5828 raft_consensus.cc:482] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:31.527943 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 (DEFAULT_TABLE table=default.Table_064299ed49e74e50a322f6cbf4878b98 [id=ed428cec915844f6bda0f7b5e9dd2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.527981 5841 leader_election.cc:288] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.162212 5846 raft_consensus.cc:482] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.185693 4639 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 507) took 2672 ms. Trace:
I20240503 02:12:31.528043 5828 raft_consensus.cc:504] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.072702 4608 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:31.528400 4639 rpcz_store.cc:276] 0503 02:12:28.513253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.037966 (+524713us) service_pool.cc:225] Handling call
0503 02:12:29.051485 (+ 13519us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.051522 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.986625 (+1935103us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.183829 (+197204us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.185323 (+ 1494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.779704 5802 raft_consensus.cc:482] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:30.966645 5791 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:31.529560 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_402b0b1faf1e44138b755f83223a5033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:31.541818 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_402b0b1faf1e44138b755f83223a5033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:31.533880 4271 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 695) took 2497 ms. Trace:
I20240503 02:12:31.529528 5791 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.542026 4271 rpcz_store.cc:276] 0503 02:12:29.036118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.950850 (+1914732us) service_pool.cc:225] Handling call
0503 02:12:30.951547 (+ 697us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:30.951762 (+ 215us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:30.951884 (+ 122us) catalog_manager.cc:2758] Locking tablets
0503 02:12:30.952031 (+ 147us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:31.382655 (+430624us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:31.382762 (+ 107us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:31.382820 (+ 58us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:31.533806 (+150986us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.980125 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.980290 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.980299 (+ 9us) write_op.cc:435] Acquired schema lock
0503 02:12:30.980303 (+ 4us) tablet.cc:660] Decoding operations
0503 02:12:30.980477 (+ 174us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.980495 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.980504 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:30.980660 (+ 156us) tablet.cc:699] Row locks acquired
0503 02:12:30.980665 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:30.980709 (+ 44us) write_op.cc:270] Start()
0503 02:12:30.980748 (+ 39us) write_op.cc:276] Timestamp: P: 1714702350980660 usec, L: 0
0503 02:12:30.980754 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.986220 (+ 5466us) log.cc:842] Serialized 756 byte log entry
0503 02:12:31.381823 (+395603us) op_driver.cc:444] REPLICATION: finished
0503 02:12:31.381959 (+ 136us) write_op.cc:301] APPLY: starting
0503 02:12:31.381979 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:31.382033 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:31.382068 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:31.382152 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:31.382212 (+ 60us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:31.382220 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:31.382308 (+ 88us) log.cc:842] Serialized 29 byte log entry
0503 02:12:31.382374 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.382396 (+ 22us) write_op.cc:454] Released schema lock
0503 02:12:31.382403 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:31.382460 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.382463 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":582,"apply.run_wall_time_us":581,"num_ops":2,"prepare.queue_time_us":27315,"prepare.run_cpu_time_us":1022,"prepare.run_wall_time_us":6187,"replication_time_us":401037}]]}
I20240503 02:12:31.542997 5791 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.973459 5796 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:31.543305 5796 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.543296 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b013c448c7de4b10bc993479d2bd7cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:31.544166 5796 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.977911 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.997876 4783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 520) took 2466 ms. Trace:
I20240503 02:12:31.000420 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:31.544704 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b013c448c7de4b10bc993479d2bd7cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:31.544713 4812 raft_consensus.cc:2454] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:31.544682 4783 rpcz_store.cc:276] 0503 02:12:28.530701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.087894 (+557193us) service_pool.cc:225] Handling call
0503 02:12:30.199918 (+1112024us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.199936 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.974724 (+774788us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.995766 (+ 21042us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.997581 (+ 1815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.528367 5846 raft_consensus.cc:504] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.000679 4859 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:31.529070 5802 raft_consensus.cc:504] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.001577 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:31.546782 4811 raft_consensus.cc:2379] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:31.002053 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:31.547153 5802 leader_election.cc:288] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:31.547276 4807 raft_consensus.cc:2379] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:30.960223 4502 rpcz_store.cc:276] 0503 02:12:28.304702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.305191 (+ 489us) service_pool.cc:225] Handling call
0503 02:12:29.076257 (+771066us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.076288 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.236408 (+1160120us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.344373 (+107965us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.344553 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.548239 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 211dfd4049684f5e8af927b2a19263b7 (DEFAULT_TABLE table=default.Table_00deed6cfb98406d84258657464b8f93 [id=8af4014e8da64c3aae761834a6683b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.507547 4267 rpcz_store.cc:276] 0503 02:12:29.035990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.946391 (+1910401us) service_pool.cc:225] Handling call
0503 02:12:30.946846 (+ 455us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:30.947075 (+ 229us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:30.947313 (+ 238us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:30.947406 (+ 93us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:30.979409 (+ 32003us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:31.507324 (+527915us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.948130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.948258 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.948266 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:30.948269 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:30.948395 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.948412 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.948418 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:30.948507 (+ 89us) tablet.cc:699] Row locks acquired
0503 02:12:30.948510 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:30.948553 (+ 43us) write_op.cc:270] Start()
0503 02:12:30.948593 (+ 40us) write_op.cc:276] Timestamp: P: 1714702350948507 usec, L: 0
0503 02:12:30.948597 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.948821 (+ 224us) log.cc:842] Serialized 405 byte log entry
0503 02:12:30.978584 (+ 29763us) op_driver.cc:444] REPLICATION: finished
0503 02:12:30.978728 (+ 144us) write_op.cc:301] APPLY: starting
0503 02:12:30.978754 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:30.978815 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:30.978868 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:30.978939 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:30.978994 (+ 55us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:30.979006 (+ 12us) write_op.cc:312] APPLY: finished
0503 02:12:30.979115 (+ 109us) log.cc:842] Serialized 23 byte log entry
0503 02:12:30.979193 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.979222 (+ 29us) write_op.cc:454] Released schema lock
0503 02:12:30.979234 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:30.979310 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:30.979316 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":651,"apply.run_wall_time_us":650,"num_ops":1,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":862,"raft.queue_time_us":4068,"raft.run_cpu_time_us":587,"raft.run_wall_time_us":587,"replication_time_us":29949}]]}
I20240503 02:12:31.002446 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:31.059808 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 54920us
I20240503 02:12:31.549289 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_670774410ede43baac53339c1d242797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:31.550287 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_670774410ede43baac53339c1d242797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:31.076694 5793 raft_consensus.cc:482] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.550586 5793 raft_consensus.cc:504] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:30.966570 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b080fc5ffc564420a0b7f2b7460ebbbb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:31.551003 4520 raft_consensus.cc:3037] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.551349 4523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:31.551614 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:30.972718 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dfd4049684f5e8af927b2a19263b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.551894 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:31.551868 5843 consensus_queue.cc:1035] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:12:31.552376 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:30.340387 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 140627us
I20240503 02:12:31.553095 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20e573d70bd46aab33466ccfc801465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.553153 4462 leader_election.cc:302] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:30.969128 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:31.553828 4529 raft_consensus.cc:2454] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:30.969432 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9327a9e5472b4eda854320c3a95f8e56" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:31.554188 4527 raft_consensus.cc:3037] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.960196 4495 rpcz_store.cc:276] 0503 02:12:28.417259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.771126 (+1353867us) service_pool.cc:225] Handling call
0503 02:12:29.771742 (+ 616us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.771774 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.241992 (+470218us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.288997 (+ 47005us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.289198 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.520704 5801 raft_consensus.cc:504] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.557492 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 6645c12ffce3474e9746249ce19a550a (DEFAULT_TABLE table=default.Table_2701a2118e9c4a24b91afbb68eebe26c [id=e1be5b08c5954211b146294a8b6e68f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.557711 5801 leader_election.cc:288] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:30.970800 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:31.557969 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6645c12ffce3474e9746249ce19a550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.557942 5793 leader_election.cc:288] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.537497 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:31.537932 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d with delete_type TABLET_DATA_DELETED (replaced by tablet b2560f50415444b8b2e497453a558d3a) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:31.538029 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:30.971158 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:30.973667 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17afd88bf6c040ad9e8d571eccd348dc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:31.558543 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f1e3b3df64a8f899b08ba8dccc8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.558538 4519 raft_consensus.cc:3037] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.975203 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b0672199fc4d538242c38616f2fd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.977861 4505 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 484) took 2648 ms. Trace:
I20240503 02:12:31.561055 4505 rpcz_store.cc:276] 0503 02:12:28.329627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.382171 (+ 52544us) service_pool.cc:225] Handling call
0503 02:12:29.076126 (+693955us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.076174 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.959557 (+1883383us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.977617 (+ 18060us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.977793 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.561267 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e3d3a356f349d0960276be05346888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.561679 4505 tablet_service.cc:1467] Processing CreateTablet for tablet f4012724bd7944fea411eb3c11b3feb7 (DEFAULT_TABLE table=default.Table_0bb92f02ee674d1f8317b3ee2e6cebb7 [id=2d4608c0683a419694b160e3be066cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.565418 5846 leader_election.cc:288] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.562170 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4012724bd7944fea411eb3c11b3feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.013424 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099b64f41e54cd7b7967c52892b2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.013732 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dfd4049684f5e8af927b2a19263b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.999652 5839 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240503 02:12:31.528661 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 456253us
I20240503 02:12:32.013968 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030dc0d3eeb445da91cfeef6886b651d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.287009 5851 raft_consensus.cc:482] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:32.014189 5851 raft_consensus.cc:504] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.014410 4608 leader_election.cc:341] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: acdefb33fe6e4a9893dd66f6252395d7
I20240503 02:12:32.014919 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40ed707ad074e0fbb0ee48c2137c661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.014948 5851 leader_election.cc:288] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.498245 5862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:31.529949 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030dc0d3eeb445da91cfeef6886b651d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.015753 5862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:32.016597 5862 leader_election.cc:288] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221), 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689)
W20240503 02:12:31.578920 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3db446d8a6984d9cb4175887f7e125bb (table default.Table_f24dcaf9ccd649c9ad2afc1764f3e6d5 [id=7a8979a4524b4879880b449477f750ed], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3db446d8a6984d9cb4175887f7e125bb consensus state has no leader
I20240503 02:12:32.017731 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07225ae9ad3c4e44ac0c5c51421b0115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.565256 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c283c08fe3453899834ff337798fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.532454 5841 raft_consensus.cc:2726] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:31.033878 4493 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 499) took 2615 ms. Trace:
I20240503 02:12:32.021080 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c48b14bc624c28a871d45e9429ee30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.021114 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1a5d3ddc24b44b921648bd94ad827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.022192 5106 tablet_bootstrap.cc:654] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.600989 4266 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 697) took 2563 ms. Trace:
I20240503 02:12:32.022348 4266 rpcz_store.cc:276] 0503 02:12:29.037086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.958427 (+1921341us) service_pool.cc:225] Handling call
0503 02:12:30.958822 (+ 395us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:30.959029 (+ 207us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:30.959233 (+ 204us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:30.959322 (+ 89us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:31.391258 (+431936us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:31.600887 (+209629us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:30.987136 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:30.987223 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:30.987228 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:30.987231 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:30.987306 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:30.987315 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:30.987319 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:30.987379 (+ 60us) tablet.cc:699] Row locks acquired
0503 02:12:30.987381 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:30.987408 (+ 27us) write_op.cc:270] Start()
0503 02:12:30.987432 (+ 24us) write_op.cc:276] Timestamp: P: 1714702350987364 usec, L: 0
0503 02:12:30.987434 (+ 2us) op_driver.cc:331] REPLICATION: starting
0503 02:12:30.987559 (+ 125us) log.cc:842] Serialized 405 byte log entry
0503 02:12:31.390004 (+402445us) op_driver.cc:444] REPLICATION: finished
0503 02:12:31.390841 (+ 837us) write_op.cc:301] APPLY: starting
0503 02:12:31.390853 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:31.390885 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:31.390923 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:31.390965 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:31.390993 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:31.390999 (+ 6us) write_op.cc:312] APPLY: finished
0503 02:12:31.391072 (+ 73us) log.cc:842] Serialized 23 byte log entry
0503 02:12:31.391118 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.391131 (+ 13us) write_op.cc:454] Released schema lock
0503 02:12:31.391137 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:31.391214 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:31.391218 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":805,"apply.run_cpu_time_us":422,"apply.run_wall_time_us":420,"num_ops":1,"prepare.queue_time_us":27329,"prepare.run_cpu_time_us":464,"prepare.run_wall_time_us":463,"replication_time_us":402555,"thread_start_us":12238,"threads_started":1}]]}
I20240503 02:12:31.538493 5828 leader_election.cc:288] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.581835 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.025185 4675 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:31.075170 4491 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 510) took 2628 ms. Trace:
I20240503 02:12:32.025701 4491 rpcz_store.cc:276] 0503 02:12:28.446664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.953275 (+2506611us) service_pool.cc:225] Handling call
0503 02:12:30.953845 (+ 570us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.953873 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.005867 (+ 51994us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.058473 (+ 52606us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.074888 (+ 16415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.577241 5159 ts_tablet_manager.cc:1888] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:32.026047 5159 log.cc:1196] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c0b9003f248b47a9952e6f995887ae0c
I20240503 02:12:32.026132 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0700d4aed530413983381d1dc4f78ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.026566 5159 ts_tablet_manager.cc:1909] T c0b9003f248b47a9952e6f995887ae0c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:32.026604 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 3478033c4ed94268a2436a2942c27a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.026997 4491 tablet_service.cc:1467] Processing CreateTablet for tablet d31e874e285c4b20b4547048a3150d4a (DEFAULT_TABLE table=default.Table_896028d24de641e4b765e2a64b27186c [id=c1434f171fe84c7fa154776740a7b6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.027401 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31e874e285c4b20b4547048a3150d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.027431 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 462) took 4584 ms. Trace:
I20240503 02:12:32.027546 5159 rpcz_store.cc:276] 0503 02:12:27.442605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.442709 (+ 104us) service_pool.cc:225] Handling call
0503 02:12:29.957736 (+2515027us) ts_tablet_manager.cc:1120] Deleting tablet c0b9003f248b47a9952e6f995887ae0c
0503 02:12:29.957745 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:30.212871 (+255126us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.576905 (+1364034us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.027382 (+450477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2514770}
I20240503 02:12:32.028044 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 461) took 4586 ms. Trace:
I20240503 02:12:32.028162 5159 rpcz_store.cc:276] 0503 02:12:27.441252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.443139 (+ 1887us) service_pool.cc:225] Handling call
0503 02:12:32.027888 (+4584749us) ts_tablet_manager.cc:1120] Deleting tablet 8241a88f5c3a47578c92343f9fb5f7cd
0503 02:12:32.027899 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.028000 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4584548}
I20240503 02:12:32.028566 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 474) took 4315 ms. Trace:
I20240503 02:12:32.028668 5159 rpcz_store.cc:276] 0503 02:12:27.713056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.713250 (+ 194us) service_pool.cc:225] Handling call
0503 02:12:32.028396 (+4315146us) ts_tablet_manager.cc:1120] Deleting tablet aaa77e6c9a494fa2bdc74c5d725624cb
0503 02:12:32.028406 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.028531 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4304430}
I20240503 02:12:32.029106 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31e874e285c4b20b4547048a3150d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.601807 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7fe2fd4286f4411aad562211e4bd77ae" } modify_external_catalogs: true
I20240503 02:12:32.029410 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00c621d795c44d0daa534261920d874a" candidate_term: 6 candidate_status { last_received { term: 5 index: 689 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:31.632313 4637 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 510) took 3048 ms. Trace:
I20240503 02:12:32.029701 4637 rpcz_store.cc:276] 0503 02:12:28.537115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.124061 (+586946us) service_pool.cc:225] Handling call
0503 02:12:30.201078 (+1077017us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.201107 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.519058 (+1317951us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.536824 (+ 17766us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.585524 (+ 48700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:30.965926 4499 rpcz_store.cc:276] 0503 02:12:28.432001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.197941 (+1765940us) service_pool.cc:225] Handling call
0503 02:12:30.198365 (+ 424us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.198402 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.310347 (+111945us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.342937 (+ 32590us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.343133 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.029951 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c0b9003f248b47a9952e6f995887ae0c (table default.Table_3b9cdd4b8f9e4449aed0a52093a299b8 [id=4dbc0d237a3d42aa8fe439bfeec4d668]) successfully deleted
W20240503 02:12:32.030202 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8241a88f5c3a47578c92343f9fb5f7cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8241a88f5c3a47578c92343f9fb5f7cd already in progress: creating tablet
I20240503 02:12:32.030309 4499 tablet_service.cc:1467] Processing CreateTablet for tablet f2c283c08fe3453899834ff337798fc2 (DEFAULT_TABLE table=default.Table_b12b7f4b0c7048f38687ebc605d55be3 [id=ff255d3f86d64fb89058519936513875]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:32.030439 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet aaa77e6c9a494fa2bdc74c5d725624cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaa77e6c9a494fa2bdc74c5d725624cb already in progress: creating tablet
I20240503 02:12:32.030719 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c283c08fe3453899834ff337798fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.595592 4782 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 532) took 1895 ms. Trace:
I20240503 02:12:32.030987 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f1e3b3df64a8f899b08ba8dccc8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.031046 4782 rpcz_store.cc:276] 0503 02:12:29.699632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.699721 (+ 89us) service_pool.cc:225] Handling call
0503 02:12:30.199170 (+499449us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.199198 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.555818 (+1356620us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.595268 (+ 39450us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.595484 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:31.755462 4260 connection.cc:535] client connection to 127.4.17.124:46689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:32.031541 4260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 7 similar messages]
I20240503 02:12:31.634604 4642 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 512) took 3094 ms. Trace:
I20240503 02:12:31.634563 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.031862 4642 rpcz_store.cc:276] 0503 02:12:28.537324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.131480 (+594156us) service_pool.cc:225] Handling call
0503 02:12:29.959804 (+828324us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.959837 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.531900 (+1572063us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.631532 (+ 99632us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.631719 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.031976 4668 raft_consensus.cc:2454] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:31.526774 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb (DEFAULT_TABLE table=default.Table_3c2d73cae23744ad8a1b256940d3a750 [id=430bf07595d347768337d56459901a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.634752 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.032289 4669 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:30.982950 4497 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 511) took 2534 ms. Trace:
I20240503 02:12:32.032421 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fcfe6e2dc844a486476bf4be80d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.032513 4497 rpcz_store.cc:276] 0503 02:12:28.448314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.960532 (+2512218us) service_pool.cc:225] Handling call
0503 02:12:30.961171 (+ 639us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.961215 (+ 44us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.968478 (+ 7263us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:30.982728 (+ 14250us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:30.982906 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.033016 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 0700d4aed530413983381d1dc4f78ef9 (DEFAULT_TABLE table=default.Table_1e4923fbee8e4eb59344e61bfd2b499f [id=565babc2a39249b2bff0b5df1461a46b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.607091 4774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 533) took 1906 ms. Trace:
I20240503 02:12:32.021027 4493 rpcz_store.cc:276] 0503 02:12:28.418270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.954040 (+1535770us) service_pool.cc:225] Handling call
0503 02:12:29.954506 (+ 466us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.954531 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:30.966861 (+1012330us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.032748 (+ 65887us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.033715 (+ 967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.033298 4774 rpcz_store.cc:276] 0503 02:12:29.699749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.700253 (+ 504us) service_pool.cc:225] Handling call
0503 02:12:30.199610 (+499357us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.199631 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.572707 (+1373076us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.604506 (+ 31799us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.606679 (+ 2173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.612411 4786 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 531) took 1917 ms. Trace:
I20240503 02:12:32.033511 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0700d4aed530413983381d1dc4f78ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.033533 4786 rpcz_store.cc:276] 0503 02:12:29.694482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.694542 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:30.198027 (+503485us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.198061 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.586353 (+1388292us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.611433 (+ 25080us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.612112 (+ 679us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:31.685437 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 134399us
W20240503 02:12:32.034003 4746 leader_election.cc:341] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:31.634943 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:31.641727 4636 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 518) took 3029 ms. Trace:
W20240503 02:12:31.435838 4590 kernel_stack_watchdog.cc:198] Thread 4608 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 197ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1c91292980 (unknown) at ??:0
@ 0x7f1c9128c9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f1c8b212904 google::LogMessage::Flush() at ??:0
@ 0x7f1c8b212a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1c996b83f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f1c996b9782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f1c996bc12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f1c996bbeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1c94b528cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f1c94b524ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1c94b3e4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f1c94b3ea2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f1c94adb51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f1c94ad9f45 kudu::rpc::Connection::ReadHandler() at ??:0
W20240503 02:12:32.034209 4746 leader_election.cc:341] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:31.625186 5802 raft_consensus.cc:2781] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.034233 4636 rpcz_store.cc:276] 0503 02:12:28.610298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.156876 (+546578us) service_pool.cc:225] Handling call
0503 02:12:29.969517 (+812641us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.969549 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.523189 (+1553640us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.636345 (+113156us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.636537 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:32.034373 4746 leader_election.cc:341] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.034428 5802 raft_consensus.cc:482] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.034540 5802 raft_consensus.cc:3037] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:32.034556 4746 leader_election.cc:341] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.034940 4746 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:31.754591 4747 connection.cc:535] client connection to 127.4.17.124:46689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:32.035274 5791 raft_consensus.cc:2781] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:31.660240 5863 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:31.663429 4638 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 514) took 3125 ms. Trace:
I20240503 02:12:31.809166 5872 raft_consensus.cc:482] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.035372 5791 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:32.035501 5791 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.035468 4638 rpcz_store.cc:276] 0503 02:12:28.537528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.145218 (+607690us) service_pool.cc:225] Handling call
0503 02:12:29.968593 (+823375us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.968641 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.511588 (+1542947us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.663045 (+151457us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.663377 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20240503 02:12:31.629266 4527 raft_consensus.cc:2454] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:32.036060 4527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.67:34503 (request call id 121) took 1071 ms. Trace:
I20240503 02:12:31.865832 5843 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:32.036175 4527 rpcz_store.cc:276] 0503 02:12:30.964569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.969293 (+ 4724us) service_pool.cc:225] Handling call
0503 02:12:32.036015 (+1066722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.036275 5843 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.037084 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60837d0810041cc883d3d8056d11bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.037106 5843 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:31.664302 4645 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 516) took 3114 ms. Trace:
I20240503 02:12:32.037537 4645 rpcz_store.cc:276] 0503 02:12:28.560321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.152442 (+592121us) service_pool.cc:225] Handling call
0503 02:12:29.969176 (+816734us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.969225 (+ 49us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.527877 (+1558652us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.663999 (+136122us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.664220 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.639052 4520 raft_consensus.cc:2454] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:32.037848 4520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.67:34503 (request call id 126) took 1072 ms. Trace:
I20240503 02:12:32.037984 4520 rpcz_store.cc:276] 0503 02:12:30.965098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.966406 (+ 1308us) service_pool.cc:225] Handling call
0503 02:12:32.037808 (+1071402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.665490 4640 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 508) took 3135 ms. Trace:
I20240503 02:12:32.038446 4640 rpcz_store.cc:276] 0503 02:12:28.530149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.101460 (+571311us) service_pool.cc:225] Handling call
0503 02:12:29.784874 (+683414us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.784910 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.514545 (+1729635us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.665302 (+150757us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.665445 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:31.639835 4519 raft_consensus.cc:2454] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:31.642864 5114 ts_tablet_manager.cc:1888] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:31.644032 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 2848c837598c49628c11374d3c875b3e (DEFAULT_TABLE table=default.Table_a7c7733ae249477b9cbd82203830f7a6 [id=eede36b209e340bd872e8b845af5020c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:31.754910 4609 connection.cc:535] client connection to 127.4.17.124:46689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:31.941924 5717 raft_consensus.cc:482] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:31.653879 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet dedead5a2bf347f6991cce8e9b4db708 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:31.661588 5103 ts_tablet_manager.cc:1398] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.116s user 0.001s sys 0.014s
I20240503 02:12:31.663591 4500 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 515) took 3133 ms. Trace:
I20240503 02:12:32.038983 5717 raft_consensus.cc:504] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:31.665122 4508 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 509) took 3212 ms. Trace:
W20240503 02:12:31.699306 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 137930us
W20240503 02:12:31.755185 4463 connection.cc:535] client connection to 127.4.17.124:46689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:32.039855 5717 leader_election.cc:288] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.004500 5873 raft_consensus.cc:482] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.044824 5115 ts_tablet_manager.cc:1888] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:32.046952 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848c837598c49628c11374d3c875b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.047201 4519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.67:34503 (request call id 127) took 1073 ms. Trace:
I20240503 02:12:32.047758 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:32.053242 4746 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:32.053346 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 478) took 4262 ms. Trace:
I20240503 02:12:32.061156 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:32.061560 5115 log.cc:1196] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/79884b3e73904c708d0d48bca85351b7
I20240503 02:12:32.061710 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:32.062100 5115 ts_tablet_manager.cc:1909] T 79884b3e73904c708d0d48bca85351b7 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:32.062175 5114 log.cc:1196] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/e05d88a5c49b43bea6d46feccaaa6920
I20240503 02:12:32.062463 4493 tablet_service.cc:1467] Processing CreateTablet for tablet f20e573d70bd46aab33466ccfc801465 (DEFAULT_TABLE table=default.Table_c569a33635544c3189f27a82ffb3d32f [id=a22896758eab4445a7df80811eca9dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.062896 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 453) took 4480 ms. Trace:
I20240503 02:12:32.063016 5115 rpcz_store.cc:276] 0503 02:12:27.582512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.583175 (+ 663us) service_pool.cc:225] Handling call
0503 02:12:30.204812 (+2621637us) ts_tablet_manager.cc:1120] Deleting tablet 79884b3e73904c708d0d48bca85351b7
0503 02:12:30.204819 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:31.527507 (+1322688us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.044628 (+517121us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.062844 (+ 18216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2610464}
I20240503 02:12:32.048466 4655 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 509) took 3511 ms. Trace:
I20240503 02:12:32.063498 5115 tablet_replica.cc:330] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:32.063526 4655 rpcz_store.cc:276] 0503 02:12:28.536994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.112119 (+575125us) service_pool.cc:225] Handling call
0503 02:12:30.198771 (+1086652us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.198797 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.024383 (+1825586us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.048207 (+ 23824us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.048396 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.056807 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099b64f41e54cd7b7967c52892b2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.062206 5873 raft_consensus.cc:504] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.062408 5103 ts_tablet_manager.cc:1430] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Tablet startup took 513ms
I20240503 02:12:32.069501 4508 rpcz_store.cc:276] 0503 02:12:28.439937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.949777 (+2509840us) service_pool.cc:225] Handling call
0503 02:12:30.950350 (+ 573us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:30.950384 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.623262 (+672878us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.650513 (+ 27251us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.652177 (+ 1664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.069962 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.070317 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.070528 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.070711 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.070863 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.071084 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d with delete_type TABLET_DATA_DELETED (replaced by tablet b2560f50415444b8b2e497453a558d3a) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.071256 4508 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.071316 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40ed707ad074e0fbb0ee48c2137c661. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:32.071413 4462 leader_election.cc:341] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.071954 5868 raft_consensus.cc:482] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:32.072114 5868 raft_consensus.cc:504] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.072432 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.072681 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20e573d70bd46aab33466ccfc801465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.072625 4519 rpcz_store.cc:276] 0503 02:12:30.965214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.971378 (+ 6164us) service_pool.cc:225] Handling call
0503 02:12:32.038776 (+1067398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.073437 4500 rpcz_store.cc:276] 0503 02:12:28.529862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.518542 (+2988680us) service_pool.cc:225] Handling call
0503 02:12:31.519075 (+ 533us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.519108 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:31.596553 (+ 77445us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.648908 (+ 52355us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:31.663522 (+ 14614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.063766 5115 raft_consensus.cc:2229] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.073757 5115 raft_consensus.cc:2258] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.074465 5115 ts_tablet_manager.cc:1875] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.078310 5114 ts_tablet_manager.cc:1909] T e05d88a5c49b43bea6d46feccaaa6920 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:32.079978 4490 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 518) took 3519 ms. Trace:
W20240503 02:12:32.072710 5103 ts_tablet_manager.cc:1432] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Trace:
0503 02:12:20.426181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.426857 (+ 676us) service_pool.cc:225] Handling call
0503 02:12:20.427233 (+ 376us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.427254 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.439493 (+ 12239us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.505279 (+ 65786us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.511696 (+ 6417us) inbound_call.cc:173] Queueing success response
0503 02:12:28.950719 (+8439023us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:31.525615 (+2574896us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:31.525827 (+ 212us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:31.525987 (+ 160us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":20053,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":241,"tablet-open.queue_time_us":8439046,"thread_start_us":109230,"threads_started":1}
I20240503 02:12:32.081704 5868 leader_election.cc:288] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.082108 5868 consensus_queue.cc:1035] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:32.082371 4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 50810.5us
I20240503 02:12:32.090236 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44707b6439b141b7b63f49f34f25b37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.091893 5103 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.089720 5873 leader_election.cc:288] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:32.096665 4260 consensus_peers.cc:485] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d -> Peer 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689): Couldn't send request to peer 145afaa7548f4b7fafb23f20276f0cb4. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.102002 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 461) took 4400 ms. Trace:
W20240503 02:12:32.102861 5400 leader_election.cc:334] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689): Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111)
I20240503 02:12:32.103255 5400 leader_election.cc:302] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c621d795c44d0daa534261920d874a; no voters: 145afaa7548f4b7fafb23f20276f0cb4, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:32.106895 5114 rpcz_store.cc:276] 0503 02:12:27.678519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.678620 (+ 101us) service_pool.cc:225] Handling call
0503 02:12:29.961024 (+2282404us) ts_tablet_manager.cc:1120] Deleting tablet e05d88a5c49b43bea6d46feccaaa6920
0503 02:12:29.961033 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:30.310474 (+349441us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:31.642018 (+1331544us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.079242 (+437224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281137}
I20240503 02:12:32.035497 5872 raft_consensus.cc:504] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.110080 4746 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:32.110581 5802 raft_consensus.cc:504] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.109531 4490 rpcz_store.cc:276] 0503 02:12:28.539086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.522594 (+2983508us) service_pool.cc:225] Handling call
0503 02:12:31.523006 (+ 412us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.523035 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.017984 (+494949us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.058369 (+ 40385us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.058574 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.111586 5159 rpcz_store.cc:276] 0503 02:12:27.790895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.791226 (+ 331us) service_pool.cc:225] Handling call
0503 02:12:32.028877 (+4237651us) ts_tablet_manager.cc:1120] Deleting tablet 0af50eb2b1c949a0a6a71e64a894e8a5
0503 02:12:32.028887 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.053285 (+ 24398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4237362}
W20240503 02:12:32.112790 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 49232.2us
W20240503 02:12:32.112819 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af50eb2b1c949a0a6a71e64a894e8a5 already in progress: creating tablet
I20240503 02:12:32.113078 5828 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:32.113790 5843 raft_consensus.cc:2781] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:32.114007 5828 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:32.114166 5828 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.115319 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 479) took 4322 ms. Trace:
I20240503 02:12:32.115407 5843 raft_consensus.cc:482] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:32.117677 5843 raft_consensus.cc:3037] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.117918 5159 rpcz_store.cc:276] 0503 02:12:27.791037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.791643 (+ 606us) service_pool.cc:225] Handling call
0503 02:12:32.112498 (+4320855us) ts_tablet_manager.cc:1120] Deleting tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3
0503 02:12:32.112512 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.113542 (+ 1030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4320624}
W20240503 02:12:32.118547 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 already in progress: creating tablet
I20240503 02:12:32.122263 5872 leader_election.cc:288] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.122982 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 482) took 4312 ms. Trace:
I20240503 02:12:32.123340 4507 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 526) took 2405 ms. Trace:
I20240503 02:12:32.123526 4507 rpcz_store.cc:276] 0503 02:12:29.694873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.527654 (+1832781us) service_pool.cc:225] Handling call
0503 02:12:31.528135 (+ 481us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.528162 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.062884 (+534722us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.100571 (+ 37687us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.100774 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.124365 5843 raft_consensus.cc:504] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.125618 4777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 543) took 1346 ms. Trace:
I20240503 02:12:32.125905 5159 rpcz_store.cc:276] 0503 02:12:27.806798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.806853 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:32.118269 (+4311416us) ts_tablet_manager.cc:1120] Deleting tablet 1ce9d746acfe4e9aaea600293470d373
0503 02:12:32.118283 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.118975 (+ 692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4311128}
I20240503 02:12:32.125872 4777 rpcz_store.cc:276] 0503 02:12:30.775939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.776017 (+ 78us) service_pool.cc:225] Handling call
0503 02:12:31.523832 (+747815us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.523859 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.068941 (+545082us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.121992 (+ 53051us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.122186 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.126437 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 480) took 4319 ms. Trace:
W20240503 02:12:32.126533 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1ce9d746acfe4e9aaea600293470d373 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce9d746acfe4e9aaea600293470d373 already in progress: creating tablet
I20240503 02:12:32.126560 5159 rpcz_store.cc:276] 0503 02:12:27.806516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.807256 (+ 740us) service_pool.cc:225] Handling call
0503 02:12:32.126255 (+4318999us) ts_tablet_manager.cc:1120] Deleting tablet 367de4ff0a6f4abfa2924a070f387cbd
0503 02:12:32.126268 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.126406 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4318710}
I20240503 02:12:32.126950 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 481) took 4320 ms. Trace:
I20240503 02:12:32.127074 5159 rpcz_store.cc:276] 0503 02:12:27.806672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.807623 (+ 951us) service_pool.cc:225] Handling call
0503 02:12:32.126789 (+4319166us) ts_tablet_manager.cc:1120] Deleting tablet 620204f7cb35442f87eae45f84beeee1
0503 02:12:32.126798 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.126909 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4318989}
I20240503 02:12:32.127475 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 491) took 4197 ms. Trace:
I20240503 02:12:32.127604 5159 rpcz_store.cc:276] 0503 02:12:27.929809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.929874 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:32.127330 (+4197456us) ts_tablet_manager.cc:1120] Deleting tablet 5a915d760ba14a31895a5dc7ec5b34fd
0503 02:12:32.127340 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.127426 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4197145}
I20240503 02:12:32.127934 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 498) took 3845 ms. Trace:
I20240503 02:12:32.128052 5159 rpcz_store.cc:276] 0503 02:12:28.282557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.282621 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:32.127802 (+3845181us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:32.127812 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.127898 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306632}
I20240503 02:12:32.128492 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 524) took 3276 ms. Trace:
I20240503 02:12:32.128621 5159 rpcz_store.cc:276] 0503 02:12:28.852298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.108609 (+256311us) service_pool.cc:225] Handling call
0503 02:12:32.128346 (+3019737us) ts_tablet_manager.cc:1120] Deleting tablet dbfe6b2b9cab4517ae7cabec4796c9ce
0503 02:12:32.128358 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.128461 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348524}
I20240503 02:12:32.128950 5159 tablet_replica.cc:330] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:32.129226 5159 raft_consensus.cc:2229] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.129419 5159 pending_rounds.cc:62] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:32.129509 5159 pending_rounds.cc:68] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7023420816503902208 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "4064ff59de474e0db3a578eae10c3fa6" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_77eae5a54c4c440a9928ae87553755be" dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" new_extra_config { } }
I20240503 02:12:32.130056 5159 raft_consensus.cc:2258] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.131436 5114 tablet_replica.cc:330] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:32.131439 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 367de4ff0a6f4abfa2924a070f387cbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 367de4ff0a6f4abfa2924a070f387cbd already in progress: creating tablet
I20240503 02:12:32.131754 5114 raft_consensus.cc:2229] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:32.132174 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 620204f7cb35442f87eae45f84beeee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 620204f7cb35442f87eae45f84beeee1 already in progress: creating tablet
W20240503 02:12:32.132704 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5a915d760ba14a31895a5dc7ec5b34fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a915d760ba14a31895a5dc7ec5b34fd already in progress: creating tablet
W20240503 02:12:32.133309 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 607e49cde2fd4aab9f44921d3c3921a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607e49cde2fd4aab9f44921d3c3921a5 already in progress: creating tablet
W20240503 02:12:32.133785 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet dbfe6b2b9cab4517ae7cabec4796c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfe6b2b9cab4517ae7cabec4796c9ce already in progress: creating tablet
I20240503 02:12:32.135021 5103 tablet_bootstrap.cc:654] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.134411 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.136102 5802 leader_election.cc:288] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.137019 5843 leader_election.cc:288] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.140887 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.140022 5114 raft_consensus.cc:2258] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.142046 4503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 517) took 3603 ms. Trace:
I20240503 02:12:32.142369 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.147699 4668 raft_consensus.cc:3037] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.149957 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6645c12ffce3474e9746249ce19a550a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:32.150238 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 135015us
I20240503 02:12:32.150385 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.152397 5114 ts_tablet_manager.cc:1875] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.155633 4527 raft_consensus.cc:2454] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:32.157078 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.158036 4669 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:32.158362 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.160336 4503 rpcz_store.cc:276] 0503 02:12:28.535963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.521002 (+2985039us) service_pool.cc:225] Handling call
0503 02:12:31.521424 (+ 422us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.521459 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.048544 (+527085us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.139632 (+ 91088us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.139782 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.168639 4504 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 516) took 3637 ms. Trace:
I20240503 02:12:32.169741 5115 ts_tablet_manager.cc:1888] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:32.169863 4504 rpcz_store.cc:276] 0503 02:12:28.531299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.519856 (+2988557us) service_pool.cc:225] Handling call
0503 02:12:31.520334 (+ 478us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.520364 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.024246 (+503882us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.168363 (+144117us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.168579 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.170468 5115 log.cc:1196] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:32.170890 5115 ts_tablet_manager.cc:1909] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:32.178628 4497 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 536) took 1393 ms. Trace:
I20240503 02:12:32.179683 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 211dfd4049684f5e8af927b2a19263b7 (DEFAULT_TABLE table=default.Table_00deed6cfb98406d84258657464b8f93 [id=8af4014e8da64c3aae761834a6683b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.179682 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb (DEFAULT_TABLE table=default.Table_3c2d73cae23744ad8a1b256940d3a750 [id=430bf07595d347768337d56459901a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.179252 4644 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 513) took 3623 ms. Trace:
I20240503 02:12:32.182260 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:32.183094 4497 rpcz_store.cc:276] 0503 02:12:30.785327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.032731 (+1247404us) service_pool.cc:225] Handling call
0503 02:12:32.033315 (+ 584us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.033337 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.160992 (+127655us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.178397 (+ 17405us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.178572 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.184186 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:32.118922 5791 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.188764 5828 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.189095 5159 ts_tablet_manager.cc:1875] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.189628 5828 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.189973 4501 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 530) took 2491 ms. Trace:
I20240503 02:12:32.190085 4501 rpcz_store.cc:276] 0503 02:12:29.698946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.629318 (+1930372us) service_pool.cc:225] Handling call
0503 02:12:32.038832 (+409514us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.046720 (+ 7888us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.157514 (+110794us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.189737 (+ 32223us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.189906 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.192274 4498 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 525) took 2498 ms. Trace:
I20240503 02:12:32.192471 4651 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 517) took 3620 ms. Trace:
I20240503 02:12:32.192406 4498 rpcz_store.cc:276] 0503 02:12:29.693623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.526433 (+1832810us) service_pool.cc:225] Handling call
0503 02:12:32.032215 (+505782us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.032243 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.141456 (+109213us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.192085 (+ 50629us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.192220 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.195623 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 (DEFAULT_TABLE table=default.Table_064299ed49e74e50a322f6cbf4878b98 [id=ed428cec915844f6bda0f7b5e9dd2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.195981 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 454) took 4588 ms. Trace:
I20240503 02:12:32.196092 5115 rpcz_store.cc:276] 0503 02:12:27.587070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.587171 (+ 101us) service_pool.cc:225] Handling call
0503 02:12:32.063409 (+4476238us) ts_tablet_manager.cc:1120] Deleting tablet b28bb9ac773c4bc685cb8193e574460b
0503 02:12:32.063424 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.112588 (+ 49164us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.169605 (+ 57017us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.171225 (+ 1620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4468837}
I20240503 02:12:32.180136 4649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57782 (request call id 511) took 3629 ms. Trace:
I20240503 02:12:32.196841 4499 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 535) took 1418 ms. Trace:
I20240503 02:12:32.196998 4499 rpcz_store.cc:276] 0503 02:12:30.778656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.030001 (+1251345us) service_pool.cc:225] Handling call
0503 02:12:32.030512 (+ 511us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.030541 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.144690 (+114149us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.196621 (+ 51931us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.196783 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.198261 4492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 514) took 3658 ms. Trace:
I20240503 02:12:32.198647 5115 tablet_replica.cc:330] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:32.198921 5115 raft_consensus.cc:2229] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.199416 5115 raft_consensus.cc:2258] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.199529 5791 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.200111 5105 tablet_bootstrap.cc:654] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.202466 4791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 542) took 1434 ms. Trace:
I20240503 02:12:32.202625 4789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39380 (request call id 544) took 1403 ms. Trace:
I20240503 02:12:32.202860 4791 rpcz_store.cc:276] 0503 02:12:30.768431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.768506 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:31.522686 (+754180us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.522726 (+ 40us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.036792 (+514066us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.202236 (+165444us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.202408 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.203718 4789 rpcz_store.cc:276] 0503 02:12:30.783585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.783691 (+ 106us) service_pool.cc:225] Handling call
0503 02:12:31.524053 (+740362us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.524077 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.110004 (+585927us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.153947 (+ 43943us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.187271 (+ 33324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.204463 4491 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 534) took 1422 ms. Trace:
I20240503 02:12:32.204985 4509 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 512) took 3743 ms. Trace:
I20240503 02:12:32.206040 5115 ts_tablet_manager.cc:1875] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.206284 4649 rpcz_store.cc:276] 0503 02:12:28.537199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.127919 (+590720us) service_pool.cc:225] Handling call
0503 02:12:29.953447 (+825528us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.953484 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.067910 (+2114426us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.166921 (+ 99011us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.167078 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.207759 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211dfd4049684f5e8af927b2a19263b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.199709 4644 rpcz_store.cc:276] 0503 02:12:28.537439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.142497 (+605058us) service_pool.cc:225] Handling call
0503 02:12:29.963478 (+820981us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.963525 (+ 47us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.018864 (+2055339us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.160358 (+141494us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.160499 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.211774 4651 rpcz_store.cc:276] 0503 02:12:28.583173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.154868 (+571695us) service_pool.cc:225] Handling call
0503 02:12:29.969636 (+814768us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:29.969657 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.110914 (+2141257us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.192085 (+ 81171us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.192429 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.213397 4644 tablet_service.cc:1467] Processing CreateTablet for tablet f4012724bd7944fea411eb3c11b3feb7 (DEFAULT_TABLE table=default.Table_0bb92f02ee674d1f8317b3ee2e6cebb7 [id=2d4608c0683a419694b160e3be066cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.213548 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fcfe6e2dc844a486476bf4be80d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.213485 4491 rpcz_store.cc:276] 0503 02:12:30.771162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.026743 (+1255581us) service_pool.cc:225] Handling call
0503 02:12:32.027193 (+ 450us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.027227 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.104318 (+ 77091us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.187022 (+ 82704us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.193307 (+ 6285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.214041 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 6645c12ffce3474e9746249ce19a550a (DEFAULT_TABLE table=default.Table_2701a2118e9c4a24b91afbb68eebe26c [id=e1be5b08c5954211b146294a8b6e68f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.221913 5862 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:12:32.222515 5419 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.224040 4639 tablet_service.cc:1467] Processing CreateTablet for tablet f2c283c08fe3453899834ff337798fc2 (DEFAULT_TABLE table=default.Table_b12b7f4b0c7048f38687ebc605d55be3 [id=ff255d3f86d64fb89058519936513875]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.224099 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 2848c837598c49628c11374d3c875b3e (DEFAULT_TABLE table=default.Table_a7c7733ae249477b9cbd82203830f7a6 [id=eede36b209e340bd872e8b845af5020c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.224208 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet dedead5a2bf347f6991cce8e9b4db708 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.224071 4637 tablet_service.cc:1467] Processing CreateTablet for tablet d31e874e285c4b20b4547048a3150d4a (DEFAULT_TABLE table=default.Table_896028d24de641e4b765e2a64b27186c [id=c1434f171fe84c7fa154776740a7b6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.224699 4492 rpcz_store.cc:276] 0503 02:12:28.528470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.516099 (+2987629us) service_pool.cc:225] Handling call
0503 02:12:31.516674 (+ 575us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.516712 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.063888 (+547176us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.186248 (+122360us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.186417 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.227034 4509 rpcz_store.cc:276] 0503 02:12:28.449997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.505950 (+3055953us) service_pool.cc:225] Handling call
0503 02:12:31.506578 (+ 628us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.506610 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.032422 (+525812us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.193527 (+161105us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.193679 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.229187 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4012724bd7944fea411eb3c11b3feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.229346 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 3478033c4ed94268a2436a2942c27a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.230768 4493 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 537) took 1441 ms. Trace:
I20240503 02:12:32.231287 4502 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 527) took 2513 ms. Trace:
I20240503 02:12:32.232779 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.232933 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.234298 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.234571 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 79884b3e73904c708d0d48bca85351b7 (table default.Table_b839b2ab5a8a4a399e63762bec827155 [id=156d16ce2a4246cd81968e098a9227dc]) successfully deleted
I20240503 02:12:32.234731 4651 tablet_service.cc:1467] Processing CreateTablet for tablet f20e573d70bd46aab33466ccfc801465 (DEFAULT_TABLE table=default.Table_c569a33635544c3189f27a82ffb3d32f [id=a22896758eab4445a7df80811eca9dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.234802 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.235247 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b28bb9ac773c4bc685cb8193e574460b (table default.Table_621dd8b829b34a0eafce9326b41c3a25 [id=01b4413eb03c47f8b8e417c80d22ae61]) successfully deleted
I20240503 02:12:32.235294 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6645c12ffce3474e9746249ce19a550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.236502 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.237335 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb69d5f4bd427a8e430a00fb076f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.238078 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.238546 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 0700d4aed530413983381d1dc4f78ef9 (DEFAULT_TABLE table=default.Table_1e4923fbee8e4eb59344e61bfd2b499f [id=565babc2a39249b2bff0b5df1461a46b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:32.233599 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 136115us
I20240503 02:12:32.238871 4652 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d with delete_type TABLET_DATA_DELETED (replaced by tablet b2560f50415444b8b2e497453a558d3a) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.238941 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.240427 5851 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:32.239924 4746 leader_election.cc:302] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:32.240790 4608 leader_election.cc:341] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:32.240999 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.241112 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c283c08fe3453899834ff337798fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.242156 4493 rpcz_store.cc:276] 0503 02:12:30.789642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.033425 (+1243783us) service_pool.cc:225] Handling call
0503 02:12:32.062689 (+ 29264us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:32.062726 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.156817 (+ 94091us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.230526 (+ 73709us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.230715 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.245040 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f1e3b3df64a8f899b08ba8dccc8f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:32.247097 4608 leader_election.cc:341] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:32.247030 5851 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:32.250463 4502 rpcz_store.cc:276] 0503 02:12:29.696130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.547805 (+1851675us) service_pool.cc:225] Handling call
0503 02:12:31.548678 (+ 873us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.548721 (+ 43us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.132835 (+584114us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.209695 (+ 76860us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.209868 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.246951 5907 raft_consensus.cc:2726] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.252712 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.253371 4505 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 529) took 2551 ms. Trace:
I20240503 02:12:32.253517 4505 rpcz_store.cc:276] 0503 02:12:29.698362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.561341 (+1862979us) service_pool.cc:225] Handling call
0503 02:12:31.561910 (+ 569us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.561939 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.226065 (+664126us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.249966 (+ 23901us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.250112 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:12:32.254132 4462 leader_election.cc:341] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.254559 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31e874e285c4b20b4547048a3150d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.255703 5851 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.262373 4746 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.263638 5907 raft_consensus.cc:2781] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.263767 5907 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.263876 5907 raft_consensus.cc:3037] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.266757 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848c837598c49628c11374d3c875b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.268251 4506 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 513) took 3815 ms. Trace:
I20240503 02:12:32.268508 4494 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 519) took 3712 ms. Trace:
I20240503 02:12:32.268931 4506 rpcz_store.cc:276] 0503 02:12:28.452899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.509461 (+3056562us) service_pool.cc:225] Handling call
0503 02:12:31.510039 (+ 578us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.510079 (+ 40us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.249907 (+739828us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.267981 (+ 18074us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.268196 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.269120 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20e573d70bd46aab33466ccfc801465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.268401 5901 raft_consensus.cc:482] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:32.269369 5901 raft_consensus.cc:504] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.271221 4494 rpcz_store.cc:276] 0503 02:12:28.540456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.523205 (+2982749us) service_pool.cc:225] Handling call
0503 02:12:31.523596 (+ 391us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.523624 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.166127 (+642503us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.252841 (+ 86714us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.253039 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.274451 5103 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.274672 5103 ts_tablet_manager.cc:1367] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.194s user 0.012s sys 0.041s
I20240503 02:12:32.274927 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0700d4aed530413983381d1dc4f78ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.275620 5103 raft_consensus.cc:348] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.275851 5103 raft_consensus.cc:374] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.275938 5103 raft_consensus.cc:729] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.276238 5103 consensus_queue.cc:260] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.278170 5901 leader_election.cc:288] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.278184 4746 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.280969 4495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 528) took 2583 ms. Trace:
I20240503 02:12:32.281101 4495 rpcz_store.cc:276] 0503 02:12:29.697558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.557097 (+1859539us) service_pool.cc:225] Handling call
0503 02:12:31.557764 (+ 667us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:31.557804 (+ 40us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:32.263177 (+705373us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.280709 (+ 17532us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:32.280909 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:32.281879 4668 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:32.283776 4462 leader_election.cc:341] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:32.284034 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.285099 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet e05d88a5c49b43bea6d46feccaaa6920 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
I20240503 02:12:32.286036 5791 raft_consensus.cc:2726] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.287626 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.288726 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.292611 5907 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.294342 5907 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.296522 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.297426 4746 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.288993 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.298472 5907 raft_consensus.cc:2726] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.299829 5106 tablet_bootstrap.cc:492] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:32.297922 4746 leader_election.cc:302] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.302965 4746 leader_election.cc:302] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.303268 5106 ts_tablet_manager.cc:1367] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.794s user 0.004s sys 0.052s
I20240503 02:12:32.311120 5791 raft_consensus.cc:2781] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.311342 5791 raft_consensus.cc:686] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:32.311648 5828 raft_consensus.cc:2781] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.311852 5828 raft_consensus.cc:686] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:32.311764 5791 consensus_queue.cc:237] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.312167 5828 consensus_queue.cc:237] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.323047 4746 leader_election.cc:302] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:32.323670 4746 leader_election.cc:341] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.326638 5105 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.326812 5105 ts_tablet_manager.cc:1367] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.815s user 0.032s sys 0.037s
I20240503 02:12:32.329054 5419 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" instance_seqno: 1714702333742995) as {username='slave'} at 127.4.17.65:40359; Asking this server to re-register.
W20240503 02:12:32.331970 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 47777.3us
W20240503 02:12:32.332435 4462 consensus_peers.cc:485] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79884b3e73904c708d0d48bca85351b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.332720 5103 ts_tablet_manager.cc:1398] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20240503 02:12:32.333093 5103 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.332674 4462 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.333609 4462 leader_election.cc:302] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.334091 4462 leader_election.cc:302] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.334597 5802 raft_consensus.cc:2781] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.334730 5802 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.334846 5802 raft_consensus.cc:3037] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.335005 5901 raft_consensus.cc:2726] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.335201 5901 raft_consensus.cc:2726] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.311321 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:32.334592 4462 leader_election.cc:341] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.336966 4462 leader_election.cc:302] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.337667 5873 raft_consensus.cc:2726] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.338814 5839 tablet.cc:1722] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:32.340167 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0d62ff4ac2460a9620a8e16e5d6548" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.340631 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe75d31e5da645be9ccdde4d0a4f9e05" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.341058 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c18cecefab4b6caed0039d09cc1890" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.348735 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da12dd28e03f462b905d1844e2ad3340" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.349579 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cdb861bc9f448698e243718136c303" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.351176 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40cb35a8d544e85ae13377724ed4727" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.351820 4575 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:32.353003 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb69d5f4bd427a8e430a00fb076f83" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.353266 4521 raft_consensus.cc:2379] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
W20240503 02:12:32.355628 4462 leader_election.cc:341] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:32.355876 4462 leader_election.cc:341] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:32.356225 4462 consensus_peers.cc:485] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79884b3e73904c708d0d48bca85351b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.360329 5843 raft_consensus.cc:2781] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.360574 5843 raft_consensus.cc:686] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:32.360683 5872 consensus_queue.cc:1035] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.361003 5843 consensus_queue.cc:237] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.361701 5901 raft_consensus.cc:482] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.361873 5901 raft_consensus.cc:504] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.362547 5901 leader_election.cc:288] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.364499 5105 raft_consensus.cc:348] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.364830 5105 raft_consensus.cc:374] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.365350 5105 raft_consensus.cc:729] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.365700 5105 consensus_queue.cc:260] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.366535 5105 ts_tablet_manager.cc:1398] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.040s user 0.001s sys 0.002s
I20240503 02:12:32.366936 5105 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:32.371217 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 47290.7us
I20240503 02:12:32.371644 4746 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:32.372555 4608 leader_election.cc:341] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: acdefb33fe6e4a9893dd66f6252395d7
I20240503 02:12:32.378999 5926 tablet.cc:1722] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:32.379451 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c18cecefab4b6caed0039d09cc1890" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.382923 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.383008 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:32.383131 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.383239 4810 raft_consensus.cc:3037] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.384214 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.386005 5105 tablet_bootstrap.cc:654] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.385944 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.388038 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.388765 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63499fb112c48238d6b84110e394779" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.389480 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9db92b88674d908b1dea62e581caac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.390185 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.391111 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.374084 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aae500e9a842ed90357f4a65fd37ba" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.393483 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:46689
--webserver_interface=127.4.17.124
--webserver_port=46343
--builtin_ntp_servers=127.4.17.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:12:32.394575 4675 raft_consensus.cc:2379] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:32.394610 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.395012 4608 leader_election.cc:302] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.395084 4810 raft_consensus.cc:2454] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:32.394717 5802 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.395433 4522 raft_consensus.cc:2379] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:32.395922 5802 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.396221 4529 raft_consensus.cc:2379] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:32.397456 5159 ts_tablet_manager.cc:1888] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:32.397634 5159 log.cc:1196] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/4064ff59de474e0db3a578eae10c3fa6
I20240503 02:12:32.398597 5159 ts_tablet_manager.cc:1909] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:32.399276 4608 raft_consensus.cc:2670] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:32.400228 5114 ts_tablet_manager.cc:1888] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:32.400358 5114 log.cc:1196] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:32.400743 5114 ts_tablet_manager.cc:1909] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:32.400877 5419 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.401005 4608 leader_election.cc:302] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.400096 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.402335 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 525) took 3549 ms. Trace:
I20240503 02:12:32.402470 5159 rpcz_store.cc:276] 0503 02:12:28.852445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.113248 (+260803us) service_pool.cc:225] Handling call
0503 02:12:32.128863 (+3015615us) ts_tablet_manager.cc:1120] Deleting tablet 4064ff59de474e0db3a578eae10c3fa6
0503 02:12:32.128873 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.225677 (+ 96804us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.397319 (+171642us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.402274 (+ 4955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347872}
I20240503 02:12:32.403008 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 526) took 3550 ms. Trace:
I20240503 02:12:32.403110 5159 rpcz_store.cc:276] 0503 02:12:28.852569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.118778 (+266209us) service_pool.cc:225] Handling call
0503 02:12:32.402806 (+3284028us) ts_tablet_manager.cc:1120] Deleting tablet 0cc2ebfb11aa4840adcc35da73be75a2
0503 02:12:32.402818 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.402957 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621562}
I20240503 02:12:32.403662 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_77eae5a54c4c440a9928ae87553755be [id=5108d90cc915403ba879db13f6a9f1be]) successfully deleted
W20240503 02:12:32.404115 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0cc2ebfb11aa4840adcc35da73be75a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc2ebfb11aa4840adcc35da73be75a2 already in progress: creating tablet
I20240503 02:12:32.404965 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbab4368e73b4944b9853c456a4db7bd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.407179 5103 tablet_bootstrap.cc:654] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.413425 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 527) took 3176 ms. Trace:
I20240503 02:12:32.413571 5159 rpcz_store.cc:276] 0503 02:12:29.227350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.227465 (+ 115us) service_pool.cc:225] Handling call
0503 02:12:32.403415 (+3175950us) ts_tablet_manager.cc:1120] Deleting tablet e05d88a5c49b43bea6d46feccaaa6920
0503 02:12:32.403427 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.403501 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2441759}
I20240503 02:12:32.413995 5159 tablet_replica.cc:330] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:32.414669 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet e05d88a5c49b43bea6d46feccaaa6920 because the tablet was not found. No further retry: Not found: Tablet not found: e05d88a5c49b43bea6d46feccaaa6920
I20240503 02:12:32.415498 5159 raft_consensus.cc:2229] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:32.415886 5159 raft_consensus.cc:2258] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.416184 5892 raft_consensus.cc:2726] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.416584 5159 ts_tablet_manager.cc:1875] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.422011 4608 leader_election.cc:302] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.422811 5892 raft_consensus.cc:2726] T fe75d31e5da645be9ccdde4d0a4f9e05 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.425916 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:32.429457 4462 leader_election.cc:341] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.432307 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.435698 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ec641e403ed843fda31dabd3920aeb68
I20240503 02:12:32.436308 4268 catalog_manager.cc:6554] ReplaceTablet: tablet ec641e403ed843fda31dabd3920aeb68 deleted and replaced by tablet d35138f04f224475af324e77aa1e0191
I20240503 02:12:32.436437 4268 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 701) took 2475 ms. Trace:
I20240503 02:12:32.436542 4268 rpcz_store.cc:276] 0503 02:12:29.961340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.508319 (+1546979us) service_pool.cc:225] Handling call
0503 02:12:32.436399 (+928080us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.603669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.603838 (+ 169us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.603848 (+ 10us) write_op.cc:435] Acquired schema lock
0503 02:12:31.603852 (+ 4us) tablet.cc:660] Decoding operations
0503 02:12:31.604051 (+ 199us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.604070 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.604076 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.604196 (+ 120us) tablet.cc:699] Row locks acquired
0503 02:12:31.604200 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:31.604244 (+ 44us) write_op.cc:270] Start()
0503 02:12:31.604286 (+ 42us) write_op.cc:276] Timestamp: P: 1714702351604186 usec, L: 0
0503 02:12:31.604292 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.604515 (+ 223us) log.cc:842] Serialized 622 byte log entry
0503 02:12:32.339317 (+734802us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.434102 (+ 94785us) write_op.cc:301] APPLY: starting
0503 02:12:32.434132 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.434207 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.434253 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.434458 (+ 205us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.434544 (+ 86us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.434557 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:32.434695 (+ 138us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.435059 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.435093 (+ 34us) write_op.cc:454] Released schema lock
0503 02:12:32.435104 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.435178 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.435181 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":124,"child_traces":[["op",{"apply.queue_time_us":94508,"apply.run_cpu_time_us":891,"apply.run_wall_time_us":1156,"num_ops":2,"prepare.queue_time_us":93502,"prepare.run_cpu_time_us":1003,"prepare.run_wall_time_us":1525,"raft.queue_time_us":70,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":382,"replication_time_us":734989,"thread_start_us":93419,"threads_started":2}]]}
I20240503 02:12:32.441287 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.439905 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4064ff59de474e0db3a578eae10c3fa6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:32.442559 4608 leader_election.cc:341] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:32.442025 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.443308 5872 raft_consensus.cc:2781] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:32.443639 5872 raft_consensus.cc:482] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:32.443794 5872 raft_consensus.cc:3037] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.444000 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:32.444510 4462 leader_election.cc:341] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.445436 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.446456 5863 tablet.cc:1722] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:32.447865 5106 raft_consensus.cc:348] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.449448 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73ec6543b9a468492f3b97772ce43f9 consensus state has no leader
I20240503 02:12:32.449429 4810 raft_consensus.cc:1226] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Rejecting Update request from peer f4f9f079230a468f90e5e3a58f254b91 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20240503 02:12:32.449652 4272 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 699) took 3333 ms. Trace:
I20240503 02:12:32.449738 4272 rpcz_store.cc:276] 0503 02:12:29.116003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.502960 (+2386957us) service_pool.cc:225] Handling call
0503 02:12:31.503707 (+ 747us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:31.504017 (+ 310us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:31.504288 (+ 271us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:31.510326 (+ 6038us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:32.449177 (+938851us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:32.449594 (+ 417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.605248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.605510 (+ 262us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.605518 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:31.605549 (+ 31us) tablet.cc:660] Decoding operations
0503 02:12:31.605756 (+ 207us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.605771 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.605779 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:31.605913 (+ 134us) tablet.cc:699] Row locks acquired
0503 02:12:31.605943 (+ 30us) write_op.cc:260] PREPARE: finished
0503 02:12:31.606002 (+ 59us) write_op.cc:270] Start()
0503 02:12:31.606039 (+ 37us) write_op.cc:276] Timestamp: P: 1714702351605925 usec, L: 0
0503 02:12:31.606063 (+ 24us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.606433 (+ 370us) log.cc:842] Serialized 405 byte log entry
0503 02:12:32.445195 (+838762us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.447227 (+ 2032us) write_op.cc:301] APPLY: starting
0503 02:12:32.447256 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.447310 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.447376 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.447456 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.447521 (+ 65us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.447534 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:32.448625 (+ 1091us) log.cc:842] Serialized 23 byte log entry
0503 02:12:32.448961 (+ 336us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.448988 (+ 27us) write_op.cc:454] Released schema lock
0503 02:12:32.448997 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.449059 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.449064 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1897,"num_ops":1,"prepare.queue_time_us":94403,"prepare.run_cpu_time_us":1050,"prepare.run_wall_time_us":1326,"replication_time_us":839049}]]}
I20240503 02:12:32.450327 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.450639 4810 raft_consensus.cc:2454] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:32.451215 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:32.451519 4810 raft_consensus.cc:2379] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:32.448261 4608 leader_election.cc:302] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.452778 4746 leader_election.cc:302] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.453469 4808 raft_consensus.cc:1226] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Rejecting Update request from peer f4f9f079230a468f90e5e3a58f254b91 for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:32.454444 5106 raft_consensus.cc:374] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.458685 5892 raft_consensus.cc:2726] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.460062 5791 raft_consensus.cc:2781] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:32.460310 5791 raft_consensus.cc:482] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:32.460481 5791 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:12:32.461325 4746 leader_election.cc:341] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.461558 4746 leader_election.cc:302] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.463308 5106 raft_consensus.cc:729] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.463637 5106 consensus_queue.cc:260] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.463984 4675 raft_consensus.cc:2454] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:32.465070 5106 ts_tablet_manager.cc:1398] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.160s user 0.001s sys 0.001s
I20240503 02:12:32.465998 5106 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:32.466073 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fcfb174825f1441f861ae6816137f855
I20240503 02:12:32.466657 4273 catalog_manager.cc:6554] ReplaceTablet: tablet fcfb174825f1441f861ae6816137f855 deleted and replaced by tablet 6a271c33588f432a801b8cbc0fb79885
I20240503 02:12:32.466825 4273 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 702) took 2505 ms. Trace:
I20240503 02:12:32.466931 4273 rpcz_store.cc:276] 0503 02:12:29.961455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.509962 (+1548507us) service_pool.cc:225] Handling call
0503 02:12:32.466759 (+956797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.606675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.606873 (+ 198us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.606880 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:31.606885 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:31.607140 (+ 255us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.607153 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.607162 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.607341 (+ 179us) tablet.cc:699] Row locks acquired
0503 02:12:31.607344 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:31.607949 (+ 605us) write_op.cc:270] Start()
0503 02:12:31.608497 (+ 548us) write_op.cc:276] Timestamp: P: 1714702351607699 usec, L: 0
0503 02:12:31.608505 (+ 8us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.630154 (+ 21649us) log.cc:842] Serialized 622 byte log entry
0503 02:12:32.445475 (+815321us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.450148 (+ 4673us) write_op.cc:301] APPLY: starting
0503 02:12:32.450168 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.450213 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.450256 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.450379 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.450440 (+ 61us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.450448 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:32.450535 (+ 87us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.465477 (+ 14942us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.465510 (+ 33us) write_op.cc:454] Released schema lock
0503 02:12:32.465526 (+ 16us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.465622 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.465626 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":86,"child_traces":[["op",{"apply.queue_time_us":4629,"apply.run_cpu_time_us":755,"apply.run_wall_time_us":15542,"num_ops":2,"prepare.queue_time_us":95447,"prepare.run_cpu_time_us":1154,"prepare.run_wall_time_us":23600,"replication_time_us":836902,"threads_started":1}]]}
I20240503 02:12:32.468485 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb174825f1441f861ae6816137f855 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a271c33588f432a801b8cbc0fb79885) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.469218 5103 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.469461 5103 ts_tablet_manager.cc:1367] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.136s user 0.004s sys 0.000s
I20240503 02:12:32.469781 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb174825f1441f861ae6816137f855 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a271c33588f432a801b8cbc0fb79885) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.470535 5103 raft_consensus.cc:348] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.470867 5103 raft_consensus.cc:374] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.470961 5103 raft_consensus.cc:729] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.471268 5103 consensus_queue.cc:260] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.471863 4608 leader_election.cc:302] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.472457 5892 raft_consensus.cc:2726] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.471906 5103 ts_tablet_manager.cc:1398] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:32.473845 4608 leader_election.cc:302] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.474813 5892 raft_consensus.cc:2726] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.477151 5843 raft_consensus.cc:2726] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.477151 5828 consensus_queue.cc:1035] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:32.478839 4746 leader_election.cc:341] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.479080 4746 leader_election.cc:302] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:32.479370 5843 raft_consensus.cc:2741] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:12:32.479882 4462 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.480434 4462 leader_election.cc:302] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.480574 4746 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.480834 5843 raft_consensus.cc:2726] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.480851 4462 leader_election.cc:302] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:32.481230 4462 leader_election.cc:341] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.481307 5802 raft_consensus.cc:2726] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.481376 4462 leader_election.cc:302] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.481530 5802 raft_consensus.cc:2726] T c63499fb112c48238d6b84110e394779 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.481700 5802 raft_consensus.cc:2726] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:12:32.481773 4462 leader_election.cc:341] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:32.481871 5802 raft_consensus.cc:2726] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.481941 4462 leader_election.cc:302] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:32.482096 4462 raft_consensus.cc:2670] T c0c18cecefab4b6caed0039d09cc1890 P f4f9f079230a468f90e5e3a58f254b91: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:32.482380 4462 leader_election.cc:302] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.482575 4746 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.482780 4462 leader_election.cc:302] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.482820 5843 raft_consensus.cc:2726] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.482868 5802 raft_consensus.cc:2726] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.483094 5802 raft_consensus.cc:2781] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.483217 5802 raft_consensus.cc:686] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:32.483337 4462 leader_election.cc:302] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.483495 5802 consensus_queue.cc:237] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.484081 5901 consensus_queue.cc:1046] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:32.484230 4746 leader_election.cc:302] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.484378 5901 raft_consensus.cc:2781] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:12:32.484416 4746 raft_consensus.cc:2670] T 4064ff59de474e0db3a578eae10c3fa6 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:32.484485 5901 raft_consensus.cc:482] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.484575 5901 raft_consensus.cc:3037] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.484746 4608 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.486315 5106 tablet_bootstrap.cc:654] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.493659 5839 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 470) took 4349 ms. Trace:
I20240503 02:12:32.493793 5839 rpcz_store.cc:276] 0503 02:12:28.143832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.144153 (+ 321us) service_pool.cc:225] Handling call
0503 02:12:32.493588 (+4349435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:28.332488 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:28.337647 (+ 5159us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:28.337659 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:28.337666 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:28.337722 (+ 56us) alter_schema_op.cc:119] START. Timestamp: 1714702348337660
0503 02:12:28.337727 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:28.338031 (+ 304us) log.cc:842] Serialized 173 byte log entry
0503 02:12:32.338681 (+4000650us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.338767 (+ 86us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:32.339264 (+ 497us) log.cc:842] Serialized 5 byte log entry
0503 02:12:32.493127 (+153863us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.493495 (+ 368us) log.cc:842] Serialized 14 byte log entry
0503 02:12:32.493543 (+ 48us) alter_schema_op.cc:65] Released schema lock
0503 02:12:32.493549 (+ 6us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"prepare.queue_time_us":175369,"prepare.run_cpu_time_us":7861,"prepare.run_wall_time_us":119156,"replication_time_us":4000926,"thread_start_us":379298,"threads_started":3,"wal-append.queue_time_us":126110,"wal-append.run_cpu_time_us":613,"wal-append.run_wall_time_us":1000685}]]}
I20240503 02:12:32.494786 5839 tablet.cc:1722] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:32.496711 5868 raft_consensus.cc:3032] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:32.496815 5103 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.496838 5868 raft_consensus.cc:729] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:32.497256 5868 consensus_queue.cc:260] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.497673 5868 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.498172 5872 raft_consensus.cc:504] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.499158 5872 leader_election.cc:288] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.503577 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 463) took 4823 ms. Trace:
I20240503 02:12:32.503754 5114 rpcz_store.cc:276] 0503 02:12:27.679869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.679921 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:32.107349 (+4427428us) ts_tablet_manager.cc:1120] Deleting tablet c0c18cecefab4b6caed0039d09cc1890
0503 02:12:32.107365 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.299230 (+191865us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.400046 (+100816us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.503519 (+103473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4427199}
I20240503 02:12:32.504038 5892 raft_consensus.cc:2726] T e40cb35a8d544e85ae13377724ed4727 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.504184 5114 tablet_replica.cc:330] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:32.504456 5114 raft_consensus.cc:2229] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.504706 5114 raft_consensus.cc:2258] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.505610 5114 ts_tablet_manager.cc:1875] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.507519 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet c0c18cecefab4b6caed0039d09cc1890 (table default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]) successfully deleted
I20240503 02:12:32.512061 5115 ts_tablet_manager.cc:1888] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:32.509709 4608 leader_election.cc:302] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:32.512990 4608 leader_election.cc:341] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: b28bb9ac773c4bc685cb8193e574460b
I20240503 02:12:32.514118 4608 leader_election.cc:302] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.515189 5115 log.cc:1196] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/acdefb33fe6e4a9893dd66f6252395d7
I20240503 02:12:32.516031 4746 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.518319 5872 raft_consensus.cc:2726] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.519455 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.519493 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedead5a2bf347f6991cce8e9b4db708" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:32.521116 4608 raft_consensus.cc:2670] T b28bb9ac773c4bc685cb8193e574460b P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:32.521337 5115 ts_tablet_manager.cc:1909] T acdefb33fe6e4a9893dd66f6252395d7 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:32.522428 5105 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.522606 5105 ts_tablet_manager.cc:1367] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.029s
I20240503 02:12:32.523540 5105 raft_consensus.cc:348] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.523798 5105 raft_consensus.cc:374] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.523885 5105 raft_consensus.cc:729] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.524171 5105 consensus_queue.cc:260] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.535651 5950 tablet.cc:1722] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 02:12:32.542332 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 59032.4us
I20240503 02:12:32.544255 4746 leader_election.cc:302] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.544683 5105 ts_tablet_manager.cc:1398] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:12:32.544855 5843 raft_consensus.cc:2726] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:12:32.542690 4462 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:32.545506 5902 raft_consensus.cc:2726] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.545748 5902 consensus_queue.cc:1046] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:12:32.546100 4462 consensus_peers.cc:485] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4064ff59de474e0db3a578eae10c3fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.546382 5902 raft_consensus.cc:3032] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:32.546463 5902 raft_consensus.cc:729] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:32.546766 5902 consensus_queue.cc:260] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.547293 5902 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.548386 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.550057 5914 mvcc.cc:205] Tried to move back new op lower bound from 7023420822489346048 to 7023420812608024576. Current Snapshot: MvccSnapshot[applied={T|T < 7023420822489346048}]
I20240503 02:12:32.550840 5945 tablet.cc:1722] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:32.552764 5103 tablet_bootstrap.cc:654] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.554211 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb174825f1441f861ae6816137f855 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a271c33588f432a801b8cbc0fb79885) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.561371 5942 tablet.cc:1722] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:32.561825 5105 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.563587 5105 tablet_bootstrap.cc:654] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.574226 5105 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.575176 5105 ts_tablet_manager.cc:1367] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240503 02:12:32.576305 5105 raft_consensus.cc:348] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.577852 5105 raft_consensus.cc:374] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.578137 5105 raft_consensus.cc:729] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.578660 5105 consensus_queue.cc:260] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.580044 5105 ts_tablet_manager.cc:1398] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:32.580740 4808 raft_consensus.cc:1226] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Rejecting Update request from peer 1f50719d94564844b15e1a9211d48a81 for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:32.581992 5105 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.584448 5105 tablet_bootstrap.cc:654] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.586792 5953 tablet.cc:1722] T 8c9db92b88674d908b1dea62e581caac P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:32.593044 5926 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 468) took 4864 ms. Trace:
I20240503 02:12:32.593210 5926 rpcz_store.cc:276] 0503 02:12:27.728117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.728544 (+ 427us) service_pool.cc:225] Handling call
0503 02:12:32.592989 (+4864445us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:27.779048 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:27.779272 (+ 224us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:27.779281 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:27.779287 (+ 6us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:27.779327 (+ 40us) alter_schema_op.cc:119] START. Timestamp: 1714702347779273
0503 02:12:27.779329 (+ 2us) op_driver.cc:331] REPLICATION: starting
0503 02:12:27.779649 (+ 320us) log.cc:842] Serialized 173 byte log entry
0503 02:12:32.338786 (+4559137us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.378935 (+ 40149us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:32.379727 (+ 792us) log.cc:842] Serialized 5 byte log entry
0503 02:12:32.592233 (+212506us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.592868 (+ 635us) log.cc:842] Serialized 14 byte log entry
0503 02:12:32.592939 (+ 71us) alter_schema_op.cc:65] Released schema lock
0503 02:12:32.592946 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40075,"prepare.queue_time_us":2376,"prepare.run_cpu_time_us":49874,"prepare.run_wall_time_us":1057618,"replication_time_us":4559443,"thread_start_us":1105569,"threads_started":3,"wal-append.queue_time_us":560070,"wal-append.run_cpu_time_us":720,"wal-append.run_wall_time_us":1004683}]]}
I20240503 02:12:32.594277 5901 raft_consensus.cc:504] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.594660 5851 raft_consensus.cc:2726] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.594763 5841 consensus_queue.cc:1046] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:32.594938 5901 leader_election.cc:288] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:32.595093 5851 raft_consensus.cc:3032] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:32.595196 5851 raft_consensus.cc:729] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:32.595436 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 455) took 5007 ms. Trace:
I20240503 02:12:32.595583 5115 rpcz_store.cc:276] 0503 02:12:27.588143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.588219 (+ 76us) service_pool.cc:225] Handling call
0503 02:12:32.196381 (+4608162us) ts_tablet_manager.cc:1120] Deleting tablet acdefb33fe6e4a9893dd66f6252395d7
0503 02:12:32.196390 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.394949 (+198559us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.504131 (+109182us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.595385 (+ 91254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4600550}
I20240503 02:12:32.595824 5863 tablet.cc:1722] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:32.596014 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 451) took 5013 ms. Trace:
I20240503 02:12:32.596132 5115 rpcz_store.cc:276] 0503 02:12:27.582295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.582923 (+ 628us) service_pool.cc:225] Handling call
0503 02:12:32.595862 (+5012939us) ts_tablet_manager.cc:1120] Deleting tablet 46560ac6d94943d2a8fb3b3904f2b5b6
0503 02:12:32.595872 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.595955 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4992245}
I20240503 02:12:32.596525 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 469) took 4731 ms. Trace:
I20240503 02:12:32.596674 5115 rpcz_store.cc:276] 0503 02:12:27.864965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.865031 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:32.596353 (+4731322us) ts_tablet_manager.cc:1120] Deleting tablet aaa77e6c9a494fa2bdc74c5d725624cb
0503 02:12:32.596361 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.596476 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4730789}
I20240503 02:12:32.595582 5851 consensus_queue.cc:260] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.597023 5115 tablet_replica.cc:330] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:32.597294 5115 raft_consensus.cc:2229] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.597504 5115 raft_consensus.cc:2258] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.598116 5115 ts_tablet_manager.cc:1875] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.605612 5851 raft_consensus.cc:3037] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.607318 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet acdefb33fe6e4a9893dd66f6252395d7 (table default.Table_b2b7945a22d649119846e9c546ddc27b [id=4308660fc24848cbb4f6674011eda2e9]) successfully deleted
W20240503 02:12:32.607658 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 because the tablet was not found. No further retry: Not found: Tablet not found: 46560ac6d94943d2a8fb3b3904f2b5b6
W20240503 02:12:32.607895 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet aaa77e6c9a494fa2bdc74c5d725624cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaa77e6c9a494fa2bdc74c5d725624cb already in progress: creating tablet
I20240503 02:12:32.610953 4267 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 707) took 2216 ms. Trace:
I20240503 02:12:32.611084 4267 rpcz_store.cc:276] 0503 02:12:30.394737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.548850 (+1154113us) service_pool.cc:225] Handling call
0503 02:12:31.599167 (+ 50317us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:31.599856 (+ 689us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.610664 (+1010808us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.610802 (+ 138us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.610908 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.631611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.631816 (+ 205us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.631848 (+ 32us) write_op.cc:435] Acquired schema lock
0503 02:12:31.631852 (+ 4us) tablet.cc:660] Decoding operations
0503 02:12:31.632056 (+ 204us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.632068 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.632077 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.632383 (+ 306us) tablet.cc:699] Row locks acquired
0503 02:12:31.632387 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:31.665862 (+ 33475us) write_op.cc:270] Start()
0503 02:12:31.665948 (+ 86us) write_op.cc:276] Timestamp: P: 1714702351632400 usec, L: 0
0503 02:12:31.665955 (+ 7us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.666272 (+ 317us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.608240 (+941968us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.608952 (+ 712us) write_op.cc:301] APPLY: starting
0503 02:12:32.608967 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.609009 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.609040 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.609178 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.609221 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.609231 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:32.609297 (+ 66us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.609339 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.609357 (+ 18us) write_op.cc:454] Released schema lock
0503 02:12:32.609364 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.609418 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.609424 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":655,"apply.run_cpu_time_us":520,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":31136,"prepare.run_cpu_time_us":1121,"prepare.run_wall_time_us":34802,"replication_time_us":942263}]]}
I20240503 02:12:32.612257 4269 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 705) took 2441 ms. Trace:
I20240503 02:12:32.592774 5159 ts_tablet_manager.cc:1888] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:32.596509 5791 raft_consensus.cc:504] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.612372 4269 rpcz_store.cc:276] 0503 02:12:30.170657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.529290 (+1358633us) service_pool.cc:225] Handling call
0503 02:12:31.600557 (+ 71267us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:31.601095 (+ 538us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.611972 (+1010877us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.612109 (+ 137us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.612209 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.666478 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.666637 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.666645 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:31.666648 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:31.666774 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.666788 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.666794 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.666908 (+ 114us) tablet.cc:699] Row locks acquired
0503 02:12:31.666911 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:31.666956 (+ 45us) write_op.cc:270] Start()
0503 02:12:31.666996 (+ 40us) write_op.cc:276] Timestamp: P: 1714702351666891 usec, L: 0
0503 02:12:31.667001 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.667212 (+ 211us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.608297 (+941085us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.609501 (+ 1204us) write_op.cc:301] APPLY: starting
0503 02:12:32.609512 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.609540 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.609563 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.609649 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.609677 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.609686 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:32.609741 (+ 55us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.609786 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.609802 (+ 16us) write_op.cc:454] Released schema lock
0503 02:12:32.609809 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.609860 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.609864 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1172,"apply.run_cpu_time_us":399,"apply.run_wall_time_us":399,"num_ops":2,"prepare.queue_time_us":64835,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":820,"replication_time_us":941282,"threads_started":1}]]}
I20240503 02:12:32.612823 5159 log.cc:1196] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/37a54873b48847c39a2b961ab5c3806c
I20240503 02:12:32.606710 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:32.613052 4808 raft_consensus.cc:3037] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.613386 5159 ts_tablet_manager.cc:1909] T 37a54873b48847c39a2b961ab5c3806c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:32.612274 5955 tablet.cc:1722] T 8c9db92b88674d908b1dea62e581caac P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:32.614784 5843 raft_consensus.cc:482] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.614974 5843 raft_consensus.cc:504] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.615625 5843 leader_election.cc:288] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.615983 4811 raft_consensus.cc:1226] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Rejecting Update request from peer f4f9f079230a468f90e5e3a58f254b91 for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:32.616276 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.616852 5791 leader_election.cc:288] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.617002 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.617054 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ad14927c1441b800a204aefc2e619" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.617087 5901 consensus_queue.cc:1046] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:32.617444 5873 raft_consensus.cc:3032] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:32.617578 5873 raft_consensus.cc:729] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:32.617847 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:32.617964 5873 consensus_queue.cc:260] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.618346 4462 leader_election.cc:341] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.618386 5873 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:32.618551 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40706400f114def9e14e6d1d7114962" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:32.618959 5106 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:32.619264 4746 leader_election.cc:341] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:32.619685 4746 leader_election.cc:341] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.619833 4746 leader_election.cc:302] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.620086 5828 raft_consensus.cc:2726] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.621831 5103 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.621977 5106 ts_tablet_manager.cc:1367] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.154s user 0.006s sys 0.018s
I20240503 02:12:32.622119 5103 ts_tablet_manager.cc:1367] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.002s
I20240503 02:12:32.622926 5103 raft_consensus.cc:348] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.623203 5103 raft_consensus.cc:374] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.623311 5103 raft_consensus.cc:729] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.623611 5103 consensus_queue.cc:260] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.623750 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7d5c9a77b56a45458afbc72f38441580"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.624497 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d5c9a77b56a45458afbc72f38441580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.624529 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_048af81d4a0e49a2bd91a97f4765a567" } new_table_name: "default.Table_23526249cb074ae5b90c63d59d47e828" modify_external_catalogs: true
I20240503 02:12:32.624636 5103 ts_tablet_manager.cc:1398] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:32.625463 5103 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.625308 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_db309e0e41744d558fd04a5e5766718f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.625895 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db309e0e41744d558fd04a5e5766718f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.627337 4271 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 706) took 2455 ms. Trace:
I20240503 02:12:32.627449 4271 rpcz_store.cc:276] 0503 02:12:30.172360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.542970 (+1370610us) service_pool.cc:225] Handling call
0503 02:12:31.602029 (+ 59059us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:31.602618 (+ 589us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.627064 (+1024446us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.627177 (+ 113us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.627278 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.667338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.667454 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.667461 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:31.667463 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:31.667560 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.667573 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.667582 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.667665 (+ 83us) tablet.cc:699] Row locks acquired
0503 02:12:31.667668 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:31.667701 (+ 33us) write_op.cc:270] Start()
0503 02:12:31.667734 (+ 33us) write_op.cc:276] Timestamp: P: 1714702351667640 usec, L: 0
0503 02:12:31.667740 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.667905 (+ 165us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.622732 (+954827us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.623303 (+ 571us) write_op.cc:301] APPLY: starting
0503 02:12:32.623332 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.623396 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.623455 (+ 59us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.623687 (+ 232us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.623729 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.623740 (+ 11us) write_op.cc:312] APPLY: finished
0503 02:12:32.623848 (+ 108us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.626949 (+ 3101us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.626976 (+ 27us) write_op.cc:454] Released schema lock
0503 02:12:32.626984 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":440,"num_ops":2,"prepare.queue_time_us":64261,"prepare.run_cpu_time_us":651,"prepare.run_wall_time_us":649,"replication_time_us":954966}]]}
I20240503 02:12:32.628190 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e2ed0b0221564739b092ebd62980fc94" } modify_external_catalogs: true
I20240503 02:12:32.628372 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e2ed0b0221564739b092ebd62980fc94" } modify_external_catalogs: true
I20240503 02:12:32.629699 5103 tablet_bootstrap.cc:654] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.636291 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:32.636687 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c9db92b88674d908b1dea62e581caac (table default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57]). Expected version 1 got 0
W20240503 02:12:32.637336 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 (table default.Table_f471b5f3094048268179ff4ff91b119f [id=f95b06fca203484c90903ffc1a2dd58f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3e6a3ca705e8482a8cd6a6fb17d93470 consensus state has no leader
I20240503 02:12:32.638193 4789 tablet_service.cc:1467] Processing CreateTablet for tablet b2560f50415444b8b2e497453a558d3a (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.638834 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2560f50415444b8b2e497453a558d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.625332 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_04781eea28204bf196d694bd2986e194" } new_table_name: "default.Table_910d49929ead4e98b7c7a3d1340a6ca1" modify_external_catalogs: true
I20240503 02:12:32.640101 4270 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 703) took 2469 ms. Trace:
I20240503 02:12:32.640210 4270 rpcz_store.cc:276] 0503 02:12:30.170402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.519599 (+1349197us) service_pool.cc:225] Handling call
0503 02:12:31.599091 (+ 79492us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:31.599883 (+ 792us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.639790 (+1039907us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.639932 (+ 142us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.640026 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.668028 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.668148 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.668156 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:31.668159 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:31.668253 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.668264 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.668272 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:31.668354 (+ 82us) tablet.cc:699] Row locks acquired
0503 02:12:31.668358 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:31.668388 (+ 30us) write_op.cc:270] Start()
0503 02:12:31.668431 (+ 43us) write_op.cc:276] Timestamp: P: 1714702351668328 usec, L: 0
0503 02:12:31.668434 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.668588 (+ 154us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.622809 (+954221us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.629273 (+ 6464us) write_op.cc:301] APPLY: starting
0503 02:12:32.629288 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.629733 (+ 445us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.629761 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.629925 (+ 164us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.639249 (+ 9324us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.639259 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:32.639356 (+ 97us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.639653 (+ 297us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.639688 (+ 35us) write_op.cc:454] Released schema lock
0503 02:12:32.639696 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.639755 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.639758 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":67200,"child_traces":[["op",{"apply.queue_time_us":6425,"apply.run_cpu_time_us":686,"apply.run_wall_time_us":10537,"mutex_wait_us":11352,"num_ops":2,"prepare.queue_time_us":67578,"prepare.run_cpu_time_us":631,"prepare.run_wall_time_us":630,"replication_time_us":954361}]]}
I20240503 02:12:32.641659 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a0e38fafc9ce4b6d89729634d5b0243d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.642239 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0e38fafc9ce4b6d89729634d5b0243d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.642603 5105 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.642766 5105 ts_tablet_manager.cc:1367] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.002s
I20240503 02:12:32.643549 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:32.644268 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3432bf1cb8f2423c8510dcf83164d243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:32.644490 4275 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_80c858c6cd0b407da24b467f885e2b76 [id=4d9a163c0c564955a47a65f799a0018c] alter complete (version 4)
W20240503 02:12:32.644888 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3432bf1cb8f2423c8510dcf83164d243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.645538 5105 raft_consensus.cc:348] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.645807 5105 raft_consensus.cc:374] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.645891 5105 raft_consensus.cc:729] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:32.646064 4275 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 89) took 2778 ms (client timeout 1000 ms). Trace:
I20240503 02:12:32.646153 5105 consensus_queue.cc:260] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.646184 4275 rpcz_store.cc:269] 0503 02:12:29.867496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.963364 (+ 95868us) service_pool.cc:225] Handling call
0503 02:12:32.646022 (+2682658us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.630319 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:31.630490 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:31.630502 (+ 12us) write_op.cc:435] Acquired schema lock
0503 02:12:31.630506 (+ 4us) tablet.cc:660] Decoding operations
0503 02:12:31.630640 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:31.630656 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:31.630664 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:31.630748 (+ 84us) tablet.cc:699] Row locks acquired
0503 02:12:31.630751 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:31.630793 (+ 42us) write_op.cc:270] Start()
0503 02:12:31.630837 (+ 44us) write_op.cc:276] Timestamp: P: 1714702351630734 usec, L: 0
0503 02:12:31.630843 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.631405 (+ 562us) log.cc:842] Serialized 356 byte log entry
0503 02:12:32.608122 (+976717us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.608317 (+ 195us) write_op.cc:301] APPLY: starting
0503 02:12:32.608338 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.608384 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.608423 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.608488 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.608536 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.608547 (+ 11us) write_op.cc:312] APPLY: finished
0503 02:12:32.608655 (+ 108us) log.cc:842] Serialized 23 byte log entry
0503 02:12:32.608733 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.608761 (+ 28us) write_op.cc:454] Released schema lock
0503 02:12:32.608770 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.608833 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.608837 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":577,"apply.run_wall_time_us":577,"num_ops":1,"prepare.queue_time_us":91749,"prepare.run_cpu_time_us":956,"prepare.run_wall_time_us":1196,"replication_time_us":955353}]]}
I20240503 02:12:32.646723 5105 ts_tablet_manager.cc:1398] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:32.646983 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_01a2313a2c52424386a931aeecfac97c" } modify_external_catalogs: true
I20240503 02:12:32.648263 5105 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.649609 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.649924 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b9003f248b47a9952e6f995887ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.637810 4495 tablet_service.cc:1467] Processing CreateTablet for tablet b2560f50415444b8b2e497453a558d3a (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.643771 5926 tablet.cc:1722] T 8c9db92b88674d908b1dea62e581caac P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 02:12:32.653018 4266 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 90) took 937 ms (client timeout 1000 ms). Trace:
W20240503 02:12:32.653148 4266 rpcz_store.cc:269] 0503 02:12:31.715190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.086404 (+371214us) service_pool.cc:225] Handling call
0503 02:12:32.652944 (+566540us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114,"reported_tablets":16}
I20240503 02:12:32.653816 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.653954 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_01a2313a2c52424386a931aeecfac97c" } modify_external_catalogs: true
I20240503 02:12:32.655743 4273 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 708) took 2257 ms. Trace:
W20240503 02:12:32.656067 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dee34be89bda450296b916163bb10c46 consensus state has no leader
I20240503 02:12:32.656260 4267 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 709) took 2247 ms. Trace:
I20240503 02:12:32.655859 4273 rpcz_store.cc:276] 0503 02:12:30.398271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.467624 (+2069353us) service_pool.cc:225] Handling call
0503 02:12:32.623602 (+155978us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.624617 (+ 1015us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:32.625437 (+ 820us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.655483 (+ 30046us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.655602 (+ 119us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.655691 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.640115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.640230 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.640237 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:32.640239 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:32.640338 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.640353 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.640358 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:32.640600 (+ 242us) tablet.cc:699] Row locks acquired
0503 02:12:32.640603 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:32.640646 (+ 43us) write_op.cc:270] Start()
0503 02:12:32.640685 (+ 39us) write_op.cc:276] Timestamp: P: 1714702352640608 usec, L: 0
0503 02:12:32.640689 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.643731 (+ 3042us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.643932 (+ 201us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:32.654525 (+ 10593us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.654782 (+ 257us) write_op.cc:301] APPLY: starting
0503 02:12:32.654801 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.654858 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.654862 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.654984 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.655022 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.655029 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:32.655177 (+ 148us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.655359 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.655391 (+ 32us) write_op.cc:454] Released schema lock
0503 02:12:32.655399 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":357677824,"child_traces":[["op",{"apply.queue_time_us":197,"num_ops":2,"prepare.queue_time_us":14171,"prepare.run_cpu_time_us":1099,"prepare.run_wall_time_us":3912,"raft.queue_time_us":1819,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":492,"replication_time_us":13808,"spinlock_wait_cycles":6473088}]]}
I20240503 02:12:32.656385 4267 rpcz_store.cc:276] 0503 02:12:30.408732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.611728 (+2202996us) service_pool.cc:225] Handling call
0503 02:12:32.624466 (+ 12738us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.624771 (+ 305us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.630247 (+ 5476us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:32.630445 (+ 198us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:32.630521 (+ 76us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:32.655793 (+ 25272us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:32.656223 (+ 430us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.644079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.644185 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.644191 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:32.644193 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:32.644298 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.644310 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.644315 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:32.644407 (+ 92us) tablet.cc:699] Row locks acquired
0503 02:12:32.644412 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:32.644459 (+ 47us) write_op.cc:270] Start()
0503 02:12:32.644498 (+ 39us) write_op.cc:276] Timestamp: P: 1714702352644418 usec, L: 0
0503 02:12:32.644502 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.644646 (+ 144us) log.cc:842] Serialized 405 byte log entry
0503 02:12:32.654605 (+ 9959us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.655244 (+ 639us) write_op.cc:301] APPLY: starting
0503 02:12:32.655260 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.655292 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.655321 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.655369 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.655400 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.655408 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:32.655494 (+ 86us) log.cc:842] Serialized 23 byte log entry
0503 02:12:32.655549 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.655568 (+ 19us) write_op.cc:454] Released schema lock
0503 02:12:32.655574 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.655624 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.655627 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":28584704,"child_traces":[["op",{"apply.queue_time_us":596,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":1,"prepare.queue_time_us":13093,"raft.queue_time_us":1276,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":10,"replication_time_us":10085,"threads_started":1}]]}
I20240503 02:12:32.658954 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 528) took 3431 ms. Trace:
I20240503 02:12:32.659125 5159 rpcz_store.cc:276] 0503 02:12:29.227491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.227712 (+ 221us) service_pool.cc:225] Handling call
0503 02:12:32.413890 (+3186178us) ts_tablet_manager.cc:1120] Deleting tablet 37a54873b48847c39a2b961ab5c3806c
0503 02:12:32.413903 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.586762 (+172859us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.592629 (+ 5867us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.658902 (+ 66273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2448451}
I20240503 02:12:32.659636 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 529) took 3430 ms. Trace:
I20240503 02:12:32.659695 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 37a54873b48847c39a2b961ab5c3806c (table default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]) successfully deleted
I20240503 02:12:32.659780 5159 rpcz_store.cc:276] 0503 02:12:29.230286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.230630 (+ 344us) service_pool.cc:225] Handling call
0503 02:12:32.659436 (+3428806us) ts_tablet_manager.cc:1120] Deleting tablet 46560ac6d94943d2a8fb3b3904f2b5b6
0503 02:12:32.659446 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.659537 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691182}
I20240503 02:12:32.660190 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 537) took 2460 ms. Trace:
I20240503 02:12:32.660305 5159 rpcz_store.cc:276] 0503 02:12:30.199432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.199494 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:32.660030 (+2460536us) ts_tablet_manager.cc:1120] Deleting tablet 3478033c4ed94268a2436a2942c27a68
0503 02:12:32.660038 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.660151 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460297}
I20240503 02:12:32.660578 5159 tablet_replica.cc:330] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:32.660557 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 46560ac6d94943d2a8fb3b3904f2b5b6 because the tablet was not found. No further retry: Not found: Tablet not found: 46560ac6d94943d2a8fb3b3904f2b5b6
I20240503 02:12:32.660801 5159 raft_consensus.cc:2229] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:32.661156 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3478033c4ed94268a2436a2942c27a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3478033c4ed94268a2436a2942c27a68 already in progress: creating tablet
I20240503 02:12:32.662197 5159 pending_rounds.cc:62] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:32.662292 5159 pending_rounds.cc:68] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7023420822489346048 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "dedead5a2bf347f6991cce8e9b4db708" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_873e48ca8c434f55927075d87ac065b1" dest_uuid: "1f50719d94564844b15e1a9211d48a81" new_extra_config { } }
I20240503 02:12:32.662685 5159 raft_consensus.cc:2258] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.664461 5159 ts_tablet_manager.cc:1875] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.666267 4651 tablet_service.cc:1467] Processing CreateTablet for tablet b2560f50415444b8b2e497453a558d3a (DEFAULT_TABLE table=default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.670177 5106 raft_consensus.cc:348] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.671034 5106 raft_consensus.cc:374] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.671181 5106 raft_consensus.cc:729] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.674335 5103 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.674597 5103 ts_tablet_manager.cc:1367] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20240503 02:12:32.675479 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1e4923fbee8e4eb59344e61bfd2b499f" } modify_external_catalogs: true
I20240503 02:12:32.675721 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1e4923fbee8e4eb59344e61bfd2b499f" } modify_external_catalogs: true
I20240503 02:12:32.675599 5103 raft_consensus.cc:348] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.675902 5103 raft_consensus.cc:374] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.675992 5103 raft_consensus.cc:729] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.676282 5103 consensus_queue.cc:260] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.676359 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4469969f6dd045b5834db5a53c4ff0cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.676964 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4469969f6dd045b5834db5a53c4ff0cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.678673 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_269475386cb84ba993cddceff2c15394"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.679193 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_269475386cb84ba993cddceff2c15394 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.689122 4808 raft_consensus.cc:2454] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:32.694808 4462 leader_election.cc:302] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:32.695533 5802 raft_consensus.cc:2781] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.695758 5802 raft_consensus.cc:686] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:32.696076 5802 consensus_queue.cc:237] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.700022 5963 tablet.cc:1717] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:32.700650 5106 consensus_queue.cc:260] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.703855 5106 ts_tablet_manager.cc:1398] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.082s user 0.001s sys 0.002s
I20240503 02:12:32.704689 5103 ts_tablet_manager.cc:1398] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240503 02:12:32.705490 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 17a24b5c83b649e79644b08253d851d6
I20240503 02:12:32.706001 5106 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:32.706219 4274 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 704) took 2535 ms. Trace:
I20240503 02:12:32.706331 4274 rpcz_store.cc:276] 0503 02:12:30.170558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.524907 (+1354349us) service_pool.cc:225] Handling call
0503 02:12:32.029296 (+504389us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.029488 (+ 192us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:32.029606 (+ 118us) catalog_manager.cc:2758] Locking tablets
0503 02:12:32.633915 (+604309us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:32.705078 (+ 71163us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:32.705208 (+ 130us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:32.705274 (+ 66us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:32.706147 (+ 873us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.675354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.675492 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.675499 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:32.675864 (+ 365us) tablet.cc:660] Decoding operations
0503 02:12:32.676039 (+ 175us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.676059 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.676064 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:32.676225 (+ 161us) tablet.cc:699] Row locks acquired
0503 02:12:32.676227 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:32.676267 (+ 40us) write_op.cc:270] Start()
0503 02:12:32.676310 (+ 43us) write_op.cc:276] Timestamp: P: 1714702352676226 usec, L: 0
0503 02:12:32.676314 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.676540 (+ 226us) log.cc:842] Serialized 756 byte log entry
0503 02:12:32.700334 (+ 23794us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.700505 (+ 171us) write_op.cc:301] APPLY: starting
0503 02:12:32.700527 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.700586 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.700651 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.700809 (+ 158us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.700870 (+ 61us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:32.700885 (+ 15us) write_op.cc:312] APPLY: finished
0503 02:12:32.701071 (+ 186us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.701187 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.701219 (+ 32us) write_op.cc:454] Released schema lock
0503 02:12:32.701230 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.701314 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.701320 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":125,"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":886,"apply.run_wall_time_us":886,"num_ops":2,"prepare.queue_time_us":40705,"prepare.run_cpu_time_us":959,"prepare.run_wall_time_us":1304,"raft.queue_time_us":1559,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":1037,"replication_time_us":23981}]]}
I20240503 02:12:32.711833 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 552dad16f30a410280c87a6c5ef0eaa4
I20240503 02:12:32.712456 4271 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 712) took 2278 ms. Trace:
I20240503 02:12:32.712579 4271 rpcz_store.cc:276] 0503 02:12:30.433531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.628005 (+2194474us) service_pool.cc:225] Handling call
0503 02:12:32.628494 (+ 489us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.628639 (+ 145us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:32.628700 (+ 61us) catalog_manager.cc:2758] Locking tablets
0503 02:12:32.638329 (+ 9629us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:32.711606 (+ 73277us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:32.711702 (+ 96us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:32.711751 (+ 49us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:32.712398 (+ 647us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.676676 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.676764 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.676769 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:32.676771 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:32.691027 (+ 14256us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.691041 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.691048 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:32.691217 (+ 169us) tablet.cc:699] Row locks acquired
0503 02:12:32.691221 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:32.691271 (+ 50us) write_op.cc:270] Start()
0503 02:12:32.691309 (+ 38us) write_op.cc:276] Timestamp: P: 1714702352691229 usec, L: 0
0503 02:12:32.691313 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.692856 (+ 1543us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x616000022cb8. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f295fb35 00007fd3f2a27bdb 00007fd3f2a03758 00007fd3f2a03005 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74
0503 02:12:32.693115 (+ 259us) log.cc:842] Serialized 725 byte log entry
0503 02:12:32.710742 (+ 17627us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.711005 (+ 263us) write_op.cc:301] APPLY: starting
0503 02:12:32.711023 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.711074 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.711076 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.711156 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.711198 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:32.711206 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:32.711295 (+ 89us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.711366 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.711400 (+ 34us) write_op.cc:454] Released schema lock
0503 02:12:32.711408 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.711468 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.711471 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":202,"apply.run_cpu_time_us":525,"apply.run_wall_time_us":524,"num_ops":2,"prepare.queue_time_us":37659,"prepare.run_cpu_time_us":1353,"prepare.run_wall_time_us":16578,"raft.queue_time_us":445,"raft.run_cpu_time_us":18,"raft.run_wall_time_us":23,"replication_time_us":19403,"spinlock_wait_cycles":2946816}]]}
I20240503 02:12:32.715097 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:32.719866 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a24b5c83b649e79644b08253d851d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.720329 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.706732 5963 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 494) took 4371 ms. Trace:
I20240503 02:12:32.709898 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a24b5c83b649e79644b08253d851d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.717576 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.733700 5106 tablet_bootstrap.cc:654] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.733983 5103 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.734727 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a24b5c83b649e79644b08253d851d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.734778 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 552dad16f30a410280c87a6c5ef0eaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.736215 5828 raft_consensus.cc:482] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.736495 5828 raft_consensus.cc:504] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.737280 5828 leader_election.cc:288] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.738705 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.739095 5103 tablet_bootstrap.cc:654] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.739076 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:32.739384 4746 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.739557 5105 tablet_bootstrap.cc:654] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:32.739997 4746 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.740165 4746 leader_election.cc:302] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.740506 5828 raft_consensus.cc:2726] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.731160 5963 rpcz_store.cc:276] 0503 02:12:28.335619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.177886 (+842267us) service_pool.cc:225] Handling call
0503 02:12:32.706683 (+3528797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.180627 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:29.180742 (+ 115us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:32.593437 (+3412695us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:32.593448 (+ 11us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:32.593505 (+ 57us) alter_schema_op.cc:119] START. Timestamp: 1714702352593478
0503 02:12:32.593511 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.593738 (+ 227us) log.cc:842] Serialized 173 byte log entry
0503 02:12:32.664331 (+ 70593us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.696070 (+ 31739us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:32.706592 (+ 10522us) log.cc:842] Serialized 94 byte log entry
0503 02:12:32.706649 (+ 57us) alter_schema_op.cc:65] Released schema lock
0503 02:12:32.706653 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31647,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":252214,"prepare.run_wall_time_us":3413697,"raft.queue_time_us":93,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":312,"replication_time_us":70792,"thread_start_us":31978,"threads_started":1}]]}
I20240503 02:12:32.770669 4269 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 710) took 2360 ms. Trace:
I20240503 02:12:32.770846 4269 rpcz_store.cc:276] 0503 02:12:30.409994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.612934 (+2202940us) service_pool.cc:225] Handling call
0503 02:12:32.625193 (+ 12259us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.626014 (+ 821us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:32.649015 (+ 23001us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:32.770382 (+121367us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:32.770525 (+ 143us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:32.770621 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.694634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.694719 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.694723 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:32.694725 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:32.706869 (+ 12144us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.706891 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.706901 (+ 10us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:32.707018 (+ 117us) tablet.cc:699] Row locks acquired
0503 02:12:32.707021 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:32.707234 (+ 213us) write_op.cc:270] Start()
0503 02:12:32.707792 (+ 558us) write_op.cc:276] Timestamp: P: 1714702352707029 usec, L: 0
0503 02:12:32.740957 (+ 33165us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.741219 (+ 262us) log.cc:842] Serialized 433 byte log entry
0503 02:12:32.769029 (+ 27810us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.769240 (+ 211us) write_op.cc:301] APPLY: starting
0503 02:12:32.769263 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.769312 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.769342 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.769506 (+ 164us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.769550 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:32.769573 (+ 23us) write_op.cc:312] APPLY: finished
0503 02:12:32.769676 (+ 103us) log.cc:842] Serialized 29 byte log entry
0503 02:12:32.769770 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.769790 (+ 20us) write_op.cc:454] Released schema lock
0503 02:12:32.769797 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.769847 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.769851 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":27522304,"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":669,"apply.run_wall_time_us":669,"num_ops":2,"prepare.queue_time_us":45103,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":46695,"raft.queue_time_us":778,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":5,"replication_time_us":28041,"spinlock_wait_cycles":8064}]]}
I20240503 02:12:32.773488 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 50a43e428b804729adf9d8a3133c90ff
I20240503 02:12:32.778028 5159 ts_tablet_manager.cc:1888] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:32.778668 5159 log.cc:1196] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/dedead5a2bf347f6991cce8e9b4db708
I20240503 02:12:32.780169 5945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:57782 (request call id 520) took 3926 ms. Trace:
I20240503 02:12:32.780608 5159 ts_tablet_manager.cc:1909] T dedead5a2bf347f6991cce8e9b4db708 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:32.781433 5745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57782: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.206s ago.
I20240503 02:12:32.781937 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c0745a55223475893edd01d904a223b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:32.781929 5745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:57782 (request call id 521) took 3928 ms. Trace:
I20240503 02:12:32.782075 5745 rpcz_store.cc:276] 0503 02:12:28.853315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.243667 (+390352us) service_pool.cc:225] Handling call
0503 02:12:32.781872 (+3538205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.252759 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:29.255622 (+ 2863us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:32.781144 (+3525522us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:32.781155 (+ 11us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:32.781349 (+ 194us) op_driver.cc:372] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.206s ago.)
0503 02:12:32.781360 (+ 11us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":8567}]]}
I20240503 02:12:32.782975 5745 mvcc.cc:205] Tried to move back new op lower bound from 7023420822489346048 to 7023420812608024576. Current Snapshot: MvccSnapshot[applied={T|T < 7023420822489346048}]
W20240503 02:12:32.783644 5745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57782: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.208s ago.
I20240503 02:12:32.780301 5945 rpcz_store.cc:276] 0503 02:12:28.853228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.242713 (+389485us) service_pool.cc:225] Handling call
0503 02:12:32.780107 (+3537394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:29.244260 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:29.245244 (+ 984us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:29.245258 (+ 14us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:29.245465 (+ 207us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:29.245822 (+ 357us) alter_schema_op.cc:119] START. Timestamp: 1714702349240563
0503 02:12:29.246075 (+ 253us) op_driver.cc:331] REPLICATION: starting
0503 02:12:29.249531 (+ 3456us) log.cc:842] Serialized 173 byte log entry
0503 02:12:32.526215 (+3276684us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.550780 (+ 24565us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:32.551324 (+ 544us) log.cc:842] Serialized 5 byte log entry
0503 02:12:32.779759 (+228435us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.780040 (+ 281us) log.cc:842] Serialized 14 byte log entry
0503 02:12:32.780077 (+ 37us) alter_schema_op.cc:65] Released schema lock
0503 02:12:32.780080 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":24484,"prepare.queue_time_us":887,"prepare.run_cpu_time_us":1683,"prepare.run_wall_time_us":8090,"replication_time_us":3280108,"spinlock_wait_cycles":394368,"thread_start_us":26567,"threads_started":3,"wal-append.queue_time_us":2531,"wal-append.run_cpu_time_us":1246,"wal-append.run_wall_time_us":1018594}]]}
I20240503 02:12:32.784406 5945 tablet.cc:1717] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:32.784688 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a43e428b804729adf9d8a3133c90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:32.784729 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a43e428b804729adf9d8a3133c90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.785419 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a43e428b804729adf9d8a3133c90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.785490 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2560f50415444b8b2e497453a558d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.787358 5106 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:32.787750 5106 ts_tablet_manager.cc:1367] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.005s
I20240503 02:12:32.788977 5106 raft_consensus.cc:348] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.790213 5106 raft_consensus.cc:374] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.790423 5106 raft_consensus.cc:729] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.791208 5106 consensus_queue.cc:260] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.793468 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3c3549c20fa149b78d707a1a111895d7" } new_table_name: "default.Table_fa770947e71d4e70a14a390ee36057db" modify_external_catalogs: true
I20240503 02:12:32.795534 5106 ts_tablet_manager.cc:1398] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:12:32.797022 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a2295477032745c3b7b2839c77679705" } modify_external_catalogs: true
I20240503 02:12:32.797138 4268 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094] alter complete (version 1)
I20240503 02:12:32.797252 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a2295477032745c3b7b2839c77679705" } modify_external_catalogs: true
I20240503 02:12:32.792881 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b6f5af3a2ab24cdbafb7e8f9a2b10aba" } new_table_name: "default.Table_cea9de5ffb3e411d90226d13e4eb39b5" modify_external_catalogs: true
I20240503 02:12:32.798177 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 41ef3c6e5e7d40249b112144975ba220 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:32.799634 5106 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:32.800206 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bba46dea649d4895bacf0532ffeb8a7f" } new_table_name: "default.Table_8b93d7e7be90441ea960cdf3b81e4fc7" modify_external_catalogs: true
I20240503 02:12:32.802973 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.803558 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.803870 5103 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.803898 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.804064 5103 ts_tablet_manager.cc:1367] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.005s
I20240503 02:12:32.804279 5105 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.804438 5105 ts_tablet_manager.cc:1367] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.000s
I20240503 02:12:32.805235 5105 raft_consensus.cc:348] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.805447 5105 raft_consensus.cc:374] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.805541 5105 raft_consensus.cc:729] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.805765 5105 consensus_queue.cc:260] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.808368 5105 ts_tablet_manager.cc:1398] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:32.809095 5105 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.809324 4271 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:40052 (request call id 1) took 3982 ms. Trace:
I20240503 02:12:32.810019 4271 rpcz_store.cc:276] 0503 02:12:28.826651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.799531 (+3972880us) service_pool.cc:225] Handling call
0503 02:12:32.809168 (+ 9637us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7b11bf 00007fd3f17ac227 00007fd3f17abf55 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.809284 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21437696}
I20240503 02:12:32.812958 5103 raft_consensus.cc:348] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.813247 5103 raft_consensus.cc:374] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.813333 5103 raft_consensus.cc:729] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.813727 5105 tablet_bootstrap.cc:654] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.816318 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_018e8f0e1c0349d08ccce34247920629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:32.816375 5103 consensus_queue.cc:260] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.816905 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_018e8f0e1c0349d08ccce34247920629 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.818729 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:32.823905 5103 ts_tablet_manager.cc:1398] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20240503 02:12:32.824366 5103 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:32.826589 5103 tablet_bootstrap.cc:654] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.828810 5105 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.829110 5105 ts_tablet_manager.cc:1367] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20240503 02:12:32.830659 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 536) took 3041 ms. Trace:
I20240503 02:12:32.830786 5159 rpcz_store.cc:276] 0503 02:12:29.789415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.789474 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:32.660521 (+2871047us) ts_tablet_manager.cc:1120] Deleting tablet dedead5a2bf347f6991cce8e9b4db708
0503 02:12:32.660528 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.774508 (+113980us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.777905 (+ 3397us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.830590 (+ 52685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460457}
I20240503 02:12:32.831171 5159 tablet_replica.cc:330] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:32.831383 5159 raft_consensus.cc:2229] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:32.831679 5159 pending_rounds.cc:62] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:32.831751 5159 pending_rounds.cc:68] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420835072081920 op_type: NO_OP noop_request { }
I20240503 02:12:32.831557 5105 raft_consensus.cc:348] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.831882 5159 raft_consensus.cc:2866] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:32.831986 5159 raft_consensus.cc:2258] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.831938 5105 raft_consensus.cc:374] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.832058 5105 raft_consensus.cc:729] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.832504 5105 consensus_queue.cc:260] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.832762 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]) successfully deleted
I20240503 02:12:32.833914 5105 ts_tablet_manager.cc:1398] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:12:32.834144 5159 ts_tablet_manager.cc:1875] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:32.834743 5105 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.837371 5105 tablet_bootstrap.cc:654] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:32.838793 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4225c1bc133d4e18a4a47fd58820ca10 consensus state has no leader
I20240503 02:12:32.841485 4634 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 98) took 1129 ms. Trace:
I20240503 02:12:32.846807 5268 master_runner.cc:186] Error getting master registration for 127.4.17.124:46689: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111),
I20240503 02:12:32.853410 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet bbab4368e73b4944b9853c456a4db7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:32.853587 5268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:12:32.855713 5106 tablet_bootstrap.cc:654] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.856128 4634 rpcz_store.cc:276] 0503 02:12:31.711648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.711739 (+ 91us) service_pool.cc:225] Handling call
0503 02:12:32.669023 (+957284us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x616000142c54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:32.670601 (+ 1578us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x616000131930. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9aa2f628 00007f1c9d42d988 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:32.680700 (+ 10099us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x616000122e54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:32.841423 (+160723us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":650624512}
I20240503 02:12:32.856307 5103 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:32.859838 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:32.860164 4274 catalog_manager.cc:5484] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:32.861928 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f3dbcbc572f4215b7f59f3750a4b9fa consensus state has no leader
I20240503 02:12:32.862116 4272 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 711) took 2451 ms. Trace:
I20240503 02:12:32.862215 4272 rpcz_store.cc:276] 0503 02:12:30.410138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.625129 (+2214991us) service_pool.cc:225] Handling call
0503 02:12:32.640045 (+ 14916us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.640290 (+ 245us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:32.640465 (+ 175us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:32.640528 (+ 63us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:32.715275 (+ 74747us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:32.862053 (+146778us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.693305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:32.693436 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:32.693443 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:32.693448 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:32.693590 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:32.693603 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:32.693608 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:32.693680 (+ 72us) tablet.cc:699] Row locks acquired
0503 02:12:32.693684 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:32.693731 (+ 47us) write_op.cc:270] Start()
0503 02:12:32.693765 (+ 34us) write_op.cc:276] Timestamp: P: 1714702352693687 usec, L: 0
0503 02:12:32.693769 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:32.693936 (+ 167us) log.cc:842] Serialized 405 byte log entry
0503 02:12:32.711586 (+ 17650us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.711752 (+ 166us) write_op.cc:301] APPLY: starting
0503 02:12:32.711764 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:32.711797 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:32.711841 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:32.711885 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:32.711923 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:32.711930 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:32.711989 (+ 59us) log.cc:842] Serialized 23 byte log entry
0503 02:12:32.712042 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.712057 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:32.712062 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:32.712107 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:32.712111 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":403,"apply.run_wall_time_us":403,"num_ops":1,"prepare.queue_time_us":52327,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":735,"raft.queue_time_us":29,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":17801}]]}
I20240503 02:12:32.862920 4639 tablet_service.cc:1467] Processing CreateTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 (DEFAULT_TABLE table=default.Table_402b0b1faf1e44138b755f83223a5033 [id=8180f95fe9024be9a952ce85a14cff10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.863445 4649 tablet_service.cc:1467] Processing CreateTablet for tablet a4a16052d43d45aeb405c5de5dafa041 (DEFAULT_TABLE table=default.Table_b013c448c7de4b10bc993479d2bd7cd8 [id=183ae13da2fb4cab9802a5cf6094bc6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.864013 4786 tablet_service.cc:1467] Processing CreateTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 (DEFAULT_TABLE table=default.Table_402b0b1faf1e44138b755f83223a5033 [id=8180f95fe9024be9a952ce85a14cff10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.864238 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 055636f6e0214d8cb77f43b232333b3f (DEFAULT_TABLE table=default.Table_3d11cc9d7e0242eeb996280bfa1f9e6a [id=81bf43f528754be9b3d9dcf4959298e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.864733 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 9522b13ac2624c9f95b05ab335ae02da (DEFAULT_TABLE table=default.Table_670774410ede43baac53339c1d242797 [id=836d664a7b8f4012b3c9373e81371dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.865180 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 6a271c33588f432a801b8cbc0fb79885 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.865729 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 055636f6e0214d8cb77f43b232333b3f (DEFAULT_TABLE table=default.Table_3d11cc9d7e0242eeb996280bfa1f9e6a [id=81bf43f528754be9b3d9dcf4959298e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.866264 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 9522b13ac2624c9f95b05ab335ae02da (DEFAULT_TABLE table=default.Table_670774410ede43baac53339c1d242797 [id=836d664a7b8f4012b3c9373e81371dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.866878 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 6a271c33588f432a801b8cbc0fb79885 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.867329 4502 tablet_service.cc:1467] Processing CreateTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 (DEFAULT_TABLE table=default.Table_402b0b1faf1e44138b755f83223a5033 [id=8180f95fe9024be9a952ce85a14cff10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.868290 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 9522b13ac2624c9f95b05ab335ae02da (DEFAULT_TABLE table=default.Table_670774410ede43baac53339c1d242797 [id=836d664a7b8f4012b3c9373e81371dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.870100 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 6a271c33588f432a801b8cbc0fb79885 (DEFAULT_TABLE table=default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.870510 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 055636f6e0214d8cb77f43b232333b3f (DEFAULT_TABLE table=default.Table_3d11cc9d7e0242eeb996280bfa1f9e6a [id=81bf43f528754be9b3d9dcf4959298e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.870649 4774 tablet_service.cc:1467] Processing CreateTablet for tablet a4a16052d43d45aeb405c5de5dafa041 (DEFAULT_TABLE table=default.Table_b013c448c7de4b10bc993479d2bd7cd8 [id=183ae13da2fb4cab9802a5cf6094bc6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.878849 5103 ts_tablet_manager.cc:1367] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240503 02:12:32.881991 5115 ts_tablet_manager.cc:1888] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:32.882150 5115 log.cc:1196] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c0c18cecefab4b6caed0039d09cc1890
I20240503 02:12:32.882550 5115 ts_tablet_manager.cc:1909] T c0c18cecefab4b6caed0039d09cc1890 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:32.884842 5106 tablet_bootstrap.cc:492] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:32.885083 5106 ts_tablet_manager.cc:1367] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.003s
I20240503 02:12:32.886022 5106 raft_consensus.cc:348] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.886243 5106 raft_consensus.cc:374] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.886312 5106 raft_consensus.cc:729] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.886677 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9522b13ac2624c9f95b05ab335ae02da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.886591 5106 consensus_queue.cc:260] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.887192 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a271c33588f432a801b8cbc0fb79885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.887642 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a16052d43d45aeb405c5de5dafa041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.888060 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab093285c39b4ce7a0bb3de9d016a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.888545 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055636f6e0214d8cb77f43b232333b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.889721 5106 ts_tablet_manager.cc:1398] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:32.893234 5106 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:32.895700 5867 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51106: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.398s ago.
I20240503 02:12:32.895843 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2560f50415444b8b2e497453a558d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.897867 5839 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 521) took 4030 ms. Trace:
I20240503 02:12:32.897943 4493 tablet_service.cc:1467] Processing CreateTablet for tablet a4a16052d43d45aeb405c5de5dafa041 (DEFAULT_TABLE table=default.Table_b013c448c7de4b10bc993479d2bd7cd8 [id=183ae13da2fb4cab9802a5cf6094bc6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.895398 5106 tablet_bootstrap.cc:654] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.901120 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055636f6e0214d8cb77f43b232333b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.901218 5867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 532) took 2701 ms. Trace:
I20240503 02:12:32.911979 5105 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.912153 5105 ts_tablet_manager.cc:1367] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.003s
I20240503 02:12:32.913098 5105 raft_consensus.cc:348] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.913316 5105 raft_consensus.cc:374] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.913396 5105 raft_consensus.cc:729] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.913619 5105 consensus_queue.cc:260] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.916052 5105 ts_tablet_manager.cc:1398] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:32.916266 5106 tablet_bootstrap.cc:492] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:32.916556 5105 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.917156 5106 ts_tablet_manager.cc:1367] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240503 02:12:32.918021 5106 raft_consensus.cc:348] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.918216 5106 raft_consensus.cc:374] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.918308 5106 raft_consensus.cc:729] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.918566 5106 consensus_queue.cc:260] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.919937 5105 tablet_bootstrap.cc:654] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.921664 5828 raft_consensus.cc:482] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.921840 5828 raft_consensus.cc:504] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.922585 5828 leader_election.cc:288] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.923169 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:32.923692 4746 leader_election.cc:341] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.924314 5106 ts_tablet_manager.cc:1398] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:12:32.926795 5106 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:32.930687 5106 tablet_bootstrap.cc:654] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.934495 5105 tablet_bootstrap.cc:492] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.934691 5105 ts_tablet_manager.cc:1367] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 02:12:32.935534 5105 raft_consensus.cc:348] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.935773 5105 raft_consensus.cc:374] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.935856 5105 raft_consensus.cc:729] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.936141 5105 consensus_queue.cc:260] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.936785 5105 ts_tablet_manager.cc:1398] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:32.937122 5105 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.938767 4782 tablet_service.cc:1467] Processing CreateTablet for tablet d35138f04f224475af324e77aa1e0191 (DEFAULT_TABLE table=default.Table_e6e97943c4dd41638a2b090765b119a3 [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.939131 5105 tablet_bootstrap.cc:654] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.939287 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35138f04f224475af324e77aa1e0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.939808 4638 tablet_service.cc:1467] Processing CreateTablet for tablet d35138f04f224475af324e77aa1e0191 (DEFAULT_TABLE table=default.Table_e6e97943c4dd41638a2b090765b119a3 [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.949294 5106 tablet_bootstrap.cc:492] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:32.949681 5106 ts_tablet_manager.cc:1367] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240503 02:12:32.950886 5106 raft_consensus.cc:348] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.951248 5106 raft_consensus.cc:374] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.951380 5106 raft_consensus.cc:729] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.951771 5106 consensus_queue.cc:260] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.952085 5105 tablet_bootstrap.cc:492] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.952103 5828 raft_consensus.cc:482] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.952277 5105 ts_tablet_manager.cc:1367] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240503 02:12:32.952268 5828 raft_consensus.cc:504] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.953159 5828 leader_election.cc:288] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:32.953133 5105 raft_consensus.cc:348] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.953382 5105 raft_consensus.cc:374] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.953491 5105 raft_consensus.cc:729] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.953688 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.953753 5105 consensus_queue.cc:260] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.954286 4746 leader_election.cc:341] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.954341 5105 ts_tablet_manager.cc:1398] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:32.954645 5105 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:32.956414 5105 tablet_bootstrap.cc:654] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.961500 5106 ts_tablet_manager.cc:1398] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:12:32.965102 5106 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:32.964838 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:32.966131 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.963411 5867 rpcz_store.cc:276] 0503 02:12:30.200115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.025967 (+1825852us) service_pool.cc:225] Handling call
0503 02:12:32.901161 (+875194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:32.031324 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:32.031410 (+ 86us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:32.881613 (+850203us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:32.881624 (+ 11us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:32.895020 (+ 13396us) op_driver.cc:372] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.398s ago.)
0503 02:12:32.895032 (+ 12us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":4873}]]}
I20240503 02:12:32.967829 5106 tablet_bootstrap.cc:654] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.968163 5873 raft_consensus.cc:482] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.968253 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:32.968348 5873 raft_consensus.cc:504] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.968199 5839 rpcz_store.cc:276] 0503 02:12:28.851483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.523490 (+2672007us) service_pool.cc:225] Handling call
0503 02:12:32.881630 (+1358140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.640274 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:31.640616 (+ 342us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:31.640628 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:31.640639 (+ 11us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:31.640692 (+ 53us) alter_schema_op.cc:119] START. Timestamp: 1714702351640655
0503 02:12:31.640697 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.641496 (+ 799us) log.cc:842] Serialized 173 byte log entry
0503 02:12:32.471277 (+829781us) op_driver.cc:444] REPLICATION: finished
0503 02:12:32.494724 (+ 23447us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:32.495200 (+ 476us) log.cc:842] Serialized 5 byte log entry
0503 02:12:32.881156 (+385956us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.881543 (+ 387us) log.cc:842] Serialized 14 byte log entry
0503 02:12:32.881593 (+ 50us) alter_schema_op.cc:65] Released schema lock
0503 02:12:32.881596 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23393,"prepare.queue_time_us":116354,"prepare.run_cpu_time_us":5608,"prepare.run_wall_time_us":34183,"raft.queue_time_us":11812,"raft.run_cpu_time_us":1186,"raft.run_wall_time_us":15063,"replication_time_us":830541,"thread_start_us":228470,"threads_started":3,"wal-append.queue_time_us":31235}]]}
I20240503 02:12:32.969089 5873 leader_election.cc:288] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:32.969395 4746 leader_election.cc:341] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:32.969532 4746 leader_election.cc:302] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:32.969803 5828 raft_consensus.cc:2726] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:32.970362 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:32.970744 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9522b13ac2624c9f95b05ab335ae02da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.970813 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:32.971438 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bb24fceb8c3e46a89b5f2be967010467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:32.972034 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb24fceb8c3e46a89b5f2be967010467 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.972805 5105 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:32.972985 5105 ts_tablet_manager.cc:1367] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
W20240503 02:12:32.973481 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a8e996d76be4dda876f4645df56c1ce consensus state has no leader
I20240503 02:12:32.973737 5105 raft_consensus.cc:348] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:32.973937 5105 raft_consensus.cc:374] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.973995 5105 raft_consensus.cc:729] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.973929 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cd19aee3b1c2424a8295717b4227f6d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:32.974220 5105 consensus_queue.cc:260] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:32.974472 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd19aee3b1c2424a8295717b4227f6d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.974695 5105 ts_tablet_manager.cc:1398] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:32.974989 5105 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:32.975622 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17afd88bf6c040ad9e8d571eccd348dc consensus state has no leader
I20240503 02:12:32.976073 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1a5f115728484ec18350f73f8ffa672b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:32.976465 5105 tablet_bootstrap.cc:654] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:32.976563 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a5f115728484ec18350f73f8ffa672b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.979219 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bcefea9ec04543dc8b079eb3822d739f" } new_table_name: "default.Table_914edf6b009344fba3c70a9fde8b6aee" modify_external_catalogs: true
I20240503 02:12:32.980671 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab093285c39b4ce7a0bb3de9d016a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.981240 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a271c33588f432a801b8cbc0fb79885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.981237 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_930427a07e994d4f8ad39f8cd1881e2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:32.981657 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a16052d43d45aeb405c5de5dafa041. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:32.981807 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_930427a07e994d4f8ad39f8cd1881e2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:32.982206 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9522b13ac2624c9f95b05ab335ae02da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.982966 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055636f6e0214d8cb77f43b232333b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.984493 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35138f04f224475af324e77aa1e0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.993180 4509 tablet_service.cc:1467] Processing CreateTablet for tablet d35138f04f224475af324e77aa1e0191 (DEFAULT_TABLE table=default.Table_e6e97943c4dd41638a2b090765b119a3 [id=e5a5a9bb92f34937b576e14ebe63c59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.993593 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 467) took 5146 ms. Trace:
I20240503 02:12:32.993672 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35138f04f224475af324e77aa1e0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.993722 5115 rpcz_store.cc:276] 0503 02:12:27.846896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.846967 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:32.596949 (+4749982us) ts_tablet_manager.cc:1120] Deleting tablet c0c18cecefab4b6caed0039d09cc1890
0503 02:12:32.596956 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.686704 (+ 89748us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.881860 (+195156us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.993507 (+111647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4727985}
I20240503 02:12:32.994167 5115 tablet_replica.cc:330] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:32.994434 5115 raft_consensus.cc:2229] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:32.994653 5115 raft_consensus.cc:2258] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:32.998566 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c0c18cecefab4b6caed0039d09cc1890 (table default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]) successfully deleted
W20240503 02:12:32.999079 4259 catalog_manager.cc:4909] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): alter failed for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]): Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.398s ago.
I20240503 02:12:32.999897 5159 ts_tablet_manager.cc:1888] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:33.000111 5159 log.cc:1196] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/9327a9e5472b4eda854320c3a95f8e56
I20240503 02:12:33.000698 5159 ts_tablet_manager.cc:1909] T 9327a9e5472b4eda854320c3a95f8e56 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:33.001022 4462 leader_election.cc:341] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.001365 4462 leader_election.cc:341] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.002342 5114 ts_tablet_manager.cc:1888] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:33.005030 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a16052d43d45aeb405c5de5dafa041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.006906 5114 log.cc:1196] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/37a54873b48847c39a2b961ab5c3806c
I20240503 02:12:33.007217 4462 leader_election.cc:302] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:33.008812 5115 ts_tablet_manager.cc:1875] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.013489 5114 ts_tablet_manager.cc:1909] T 37a54873b48847c39a2b961ab5c3806c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:33.013674 5873 raft_consensus.cc:2726] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.014063 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a271c33588f432a801b8cbc0fb79885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.022378 4268 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173] alter complete (version 2)
I20240503 02:12:33.025913 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7e6b24685afa41e388162e59492f00b0" } modify_external_catalogs: true
I20240503 02:12:33.026136 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7e6b24685afa41e388162e59492f00b0" } modify_external_catalogs: true
I20240503 02:12:33.034183 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0700d4aed530413983381d1dc4f78ef9
I20240503 02:12:33.035039 5103 raft_consensus.cc:348] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.035320 5103 raft_consensus.cc:374] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.035733 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 0700d4aed530413983381d1dc4f78ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.036861 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.037339 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.037653 5103 raft_consensus.cc:729] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.037735 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4c0745a55223475893edd01d904a223b
I20240503 02:12:33.037946 5103 consensus_queue.cc:260] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.038286 4269 catalog_manager.cc:6554] ReplaceTablet: tablet 4c0745a55223475893edd01d904a223b deleted and replaced by tablet 3c421694ce1e424da7953d061a6bca3e
I20240503 02:12:33.039288 5103 ts_tablet_manager.cc:1398] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.160s user 0.000s sys 0.003s
I20240503 02:12:33.039737 5103 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.039222 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0745a55223475893edd01d904a223b with delete_type TABLET_DATA_DELETED (replaced by tablet 3c421694ce1e424da7953d061a6bca3e) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.040815 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab093285c39b4ce7a0bb3de9d016a918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.042374 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 0700d4aed530413983381d1dc4f78ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.044464 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0745a55223475893edd01d904a223b with delete_type TABLET_DATA_DELETED (replaced by tablet 3c421694ce1e424da7953d061a6bca3e) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.044982 4529 raft_consensus.cc:2454] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:33.045714 4746 leader_election.cc:302] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.046258 5828 raft_consensus.cc:2781] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.046370 5828 raft_consensus.cc:482] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.046494 5828 raft_consensus.cc:3037] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.047034 5873 raft_consensus.cc:482] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.047231 5873 raft_consensus.cc:504] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.048846 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b5f5d5bd4a4fd6a0cfcfb8cc9e924c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.049158 4808 raft_consensus.cc:2454] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:33.051656 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b5f5d5bd4a4fd6a0cfcfb8cc9e924c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.056984 5873 leader_election.cc:288] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.058051 4462 leader_election.cc:302] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:33.058566 4462 leader_election.cc:341] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.059108 5873 raft_consensus.cc:2781] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.059262 5873 raft_consensus.cc:482] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.059360 5873 raft_consensus.cc:3037] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.062517 5103 tablet_bootstrap.cc:654] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.063930 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0745a55223475893edd01d904a223b with delete_type TABLET_DATA_DELETED (replaced by tablet 3c421694ce1e424da7953d061a6bca3e) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.064381 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 0700d4aed530413983381d1dc4f78ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.067873 5106 tablet_bootstrap.cc:492] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.068059 5106 ts_tablet_manager.cc:1367] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.041s
I20240503 02:12:33.068845 5106 raft_consensus.cc:348] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.069096 5106 raft_consensus.cc:374] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.069200 5106 raft_consensus.cc:729] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.069473 5106 consensus_queue.cc:260] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.072479 5828 raft_consensus.cc:504] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.073257 5828 leader_election.cc:288] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.074410 5106 ts_tablet_manager.cc:1398] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:33.075701 5106 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.079777 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:33.079977 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:33.080346 4529 raft_consensus.cc:3037] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.081727 4267 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 717) took 1064 ms. Trace:
I20240503 02:12:33.081848 4267 rpcz_store.cc:276] 0503 02:12:32.017531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.657216 (+639685us) service_pool.cc:225] Handling call
0503 02:12:32.675412 (+ 18196us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.675828 (+ 416us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.675953 (+ 125us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:32.676021 (+ 68us) catalog_manager.cc:2758] Locking tablets
0503 02:12:32.857533 (+181512us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:33.033953 (+176420us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:33.034059 (+ 106us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:33.034109 (+ 50us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:33.081650 (+ 47541us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:33.019140 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:33.019246 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:33.019252 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:33.019254 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:33.019371 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:33.019382 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:33.019395 (+ 13us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:33.019493 (+ 98us) tablet.cc:699] Row locks acquired
0503 02:12:33.019496 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:33.019532 (+ 36us) write_op.cc:270] Start()
0503 02:12:33.019603 (+ 71us) write_op.cc:276] Timestamp: P: 1714702353019495 usec, L: 0
0503 02:12:33.019606 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:33.019769 (+ 163us) log.cc:842] Serialized 725 byte log entry
0503 02:12:33.033135 (+ 13366us) op_driver.cc:444] REPLICATION: finished
0503 02:12:33.033292 (+ 157us) write_op.cc:301] APPLY: starting
0503 02:12:33.033310 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:33.033375 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:33.033408 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:33.033482 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:33.033520 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:33.033528 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:33.033603 (+ 75us) log.cc:842] Serialized 29 byte log entry
0503 02:12:33.033659 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:33.033696 (+ 37us) write_op.cc:454] Released schema lock
0503 02:12:33.033703 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:33.033761 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:33.033765 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":46924,"spinlock_wait_cycles":41147264,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":527,"apply.run_wall_time_us":526,"num_ops":2,"prepare.queue_time_us":160813,"prepare.run_cpu_time_us":768,"prepare.run_wall_time_us":8356,"raft.queue_time_us":1320,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":13502}]]}
I20240503 02:12:33.082901 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.083545 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:33.085459 4746 leader_election.cc:341] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.097105 5851 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.097298 5851 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.098003 5851 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.098870 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.099148 4808 raft_consensus.cc:2379] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:33.100140 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.100462 4525 raft_consensus.cc:2379] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:33.101235 4608 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:33.101696 5851 raft_consensus.cc:3037] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.102697 5828 consensus_queue.cc:1035] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.113178 5103 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.113433 5103 ts_tablet_manager.cc:1367] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.003s
I20240503 02:12:33.114496 5103 raft_consensus.cc:348] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.114732 5103 raft_consensus.cc:374] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.114805 5103 raft_consensus.cc:729] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.115090 5103 consensus_queue.cc:260] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.116159 5103 ts_tablet_manager.cc:1398] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.116631 5103 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.119506 5103 tablet_bootstrap.cc:654] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.139019 5103 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.139225 5103 ts_tablet_manager.cc:1367] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.007s
I20240503 02:12:33.140451 5103 raft_consensus.cc:348] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.140690 5103 raft_consensus.cc:374] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.140789 5103 raft_consensus.cc:729] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.141129 5103 consensus_queue.cc:260] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.142045 5103 ts_tablet_manager.cc:1398] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.142416 5103 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.144573 5103 tablet_bootstrap.cc:654] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.156406 5103 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.156664 5103 ts_tablet_manager.cc:1367] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240503 02:12:33.157639 5103 raft_consensus.cc:348] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.157908 5103 raft_consensus.cc:374] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.158082 5103 raft_consensus.cc:729] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.158461 5103 consensus_queue.cc:260] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.159619 5103 ts_tablet_manager.cc:1398] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:12:33.160307 5103 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:33.161577 5904 log.cc:924] Time spent T 00000000000000000000000000000000 P 00c621d795c44d0daa534261920d874a: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240503 02:12:33.162129 4522 raft_consensus.cc:2454] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 2.
W20240503 02:12:33.163980 5984 log.cc:924] Time spent T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240503 02:12:33.164244 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 541) took 2916 ms. Trace:
I20240503 02:12:33.164355 5159 rpcz_store.cc:276] 0503 02:12:30.247417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.248970 (+ 1553us) service_pool.cc:225] Handling call
0503 02:12:32.831079 (+2582109us) ts_tablet_manager.cc:1120] Deleting tablet 9327a9e5472b4eda854320c3a95f8e56
0503 02:12:32.831088 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.886206 (+ 55118us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.999772 (+113566us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.164174 (+164402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2581891}
I20240503 02:12:33.164757 5106 tablet_bootstrap.cc:654] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.165023 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 9327a9e5472b4eda854320c3a95f8e56 (table default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]) successfully deleted
I20240503 02:12:33.174069 5851 raft_consensus.cc:2726] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.177441 5106 tablet_bootstrap.cc:492] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.177755 5106 ts_tablet_manager.cc:1367] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.004s
I20240503 02:12:33.178607 5106 raft_consensus.cc:348] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.178941 5106 raft_consensus.cc:374] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.179056 5106 raft_consensus.cc:729] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.179440 5106 consensus_queue.cc:260] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.182626 5115 ts_tablet_manager.cc:1888] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:33.182772 5115 log.cc:1196] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/37a54873b48847c39a2b961ab5c3806c
I20240503 02:12:33.183224 5115 ts_tablet_manager.cc:1909] T 37a54873b48847c39a2b961ab5c3806c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:33.183645 5106 ts_tablet_manager.cc:1398] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:33.186067 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 462) took 5506 ms. Trace:
I20240503 02:12:33.186192 5114 rpcz_store.cc:276] 0503 02:12:27.679722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.679941 (+ 219us) service_pool.cc:225] Handling call
0503 02:12:32.504118 (+4824177us) ts_tablet_manager.cc:1120] Deleting tablet 37a54873b48847c39a2b961ab5c3806c
0503 02:12:32.504128 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:32.683180 (+179052us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:32.989058 (+305878us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.185995 (+196937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4823503}
I20240503 02:12:33.186647 5114 tablet_replica.cc:330] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:33.186714 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 37a54873b48847c39a2b961ab5c3806c (table default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]) successfully deleted
I20240503 02:12:33.186846 5114 raft_consensus.cc:2229] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:33.187240 5114 pending_rounds.cc:62] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Trying to abort 1 pending ops.
I20240503 02:12:33.187317 5114 pending_rounds.cc:68] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420816296054784 op_type: NO_OP noop_request { }
I20240503 02:12:33.187469 5114 raft_consensus.cc:2866] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:33.187579 5114 raft_consensus.cc:2258] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.187999 5106 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.188217 5114 ts_tablet_manager.cc:1875] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.164896 5159 tablet_replica.cc:330] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:33.190246 5159 raft_consensus.cc:2229] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:33.190476 5159 raft_consensus.cc:2258] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.190732 5106 tablet_bootstrap.cc:654] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.191040 5159 ts_tablet_manager.cc:1875] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.170210 4746 leader_election.cc:302] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.193325 5828 raft_consensus.cc:2781] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:33.170786 5105 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.193853 5105 ts_tablet_manager.cc:1367] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.219s user 0.022s sys 0.041s
I20240503 02:12:33.194656 5105 raft_consensus.cc:348] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.194875 5105 raft_consensus.cc:374] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.194972 5105 raft_consensus.cc:729] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.195194 5105 consensus_queue.cc:260] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.197274 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 46005e39ab284da6bf1bb9565318a2f1
I20240503 02:12:33.197966 4270 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 719) took 1180 ms. Trace:
I20240503 02:12:33.198370 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.198082 4270 rpcz_store.cc:276] 0503 02:12:32.017675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.770206 (+752531us) service_pool.cc:225] Handling call
0503 02:12:32.793401 (+ 23195us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:32.797374 (+ 3973us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:32.797553 (+ 179us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:32.797635 (+ 82us) catalog_manager.cc:2758] Locking tablets
0503 02:12:32.861094 (+ 63459us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:33.196991 (+335897us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:33.197109 (+ 118us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:33.197174 (+ 65us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:33.197919 (+ 745us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:33.086154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:33.086270 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:33.086276 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:33.086279 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:33.086395 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:33.086410 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:33.086415 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:33.086543 (+ 128us) tablet.cc:699] Row locks acquired
0503 02:12:33.086546 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:33.086581 (+ 35us) write_op.cc:270] Start()
0503 02:12:33.086617 (+ 36us) write_op.cc:276] Timestamp: P: 1714702353086539 usec, L: 0
0503 02:12:33.086620 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:33.086826 (+ 206us) log.cc:842] Serialized 725 byte log entry
0503 02:12:33.196238 (+109412us) op_driver.cc:444] REPLICATION: finished
0503 02:12:33.196374 (+ 136us) write_op.cc:301] APPLY: starting
0503 02:12:33.196396 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:33.196458 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:33.196496 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:33.196585 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:33.196624 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:33.196633 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:33.196736 (+ 103us) log.cc:842] Serialized 29 byte log entry
0503 02:12:33.196822 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:33.196848 (+ 26us) write_op.cc:454] Released schema lock
0503 02:12:33.196859 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:33.196955 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:33.196962 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":52762368,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":647,"apply.run_wall_time_us":646,"num_ops":2,"prepare.queue_time_us":224454,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":4970,"raft.queue_time_us":2763,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":379,"replication_time_us":109589}]]}
I20240503 02:12:33.200174 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.201783 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.206296 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6e97943c4dd41638a2b090765b119a3" } modify_external_catalogs: true
I20240503 02:12:33.206655 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6e97943c4dd41638a2b090765b119a3" } modify_external_catalogs: true
W20240503 02:12:33.207660 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 335f896171b54b67ad44cb8e161003ce consensus state has no leader
I20240503 02:12:33.208007 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 8195767d80384a6c9a4690292f6776de from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:33.209587 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8eefedb5829e47eaba1cf863483cd06d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.210140 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eefedb5829e47eaba1cf863483cd06d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.211694 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_885ac00da36e484f81d9bdba48c6e155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.212366 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_885ac00da36e484f81d9bdba48c6e155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.213301 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 818788a24cd1419980c0d622fb05683b
I20240503 02:12:33.214270 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a6d78e7a628d4b7b939e74d32d5ce601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.214937 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6d78e7a628d4b7b939e74d32d5ce601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.215106 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f5be79fd35347a7ac91d7cc508b4a67" } modify_external_catalogs: true
I20240503 02:12:33.215157 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 818788a24cd1419980c0d622fb05683b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.215286 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f5be79fd35347a7ac91d7cc508b4a67" } modify_external_catalogs: true
I20240503 02:12:33.215435 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 818788a24cd1419980c0d622fb05683b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.216719 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 818788a24cd1419980c0d622fb05683b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.208835 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 95681e67db1e4485876d530133e8a514 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:33.218851 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.219758 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.220031 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.220172 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.220402 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a54873b48847c39a2b961ab5c3806c with delete_type TABLET_DATA_DELETED (replaced by tablet b3223b35d26d425582e61f70f17fbdc9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.220620 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet dedead5a2bf347f6991cce8e9b4db708 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.220532 5873 raft_consensus.cc:504] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.220809 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.221786 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b5f5d5bd4a4fd6a0cfcfb8cc9e924c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:33.222092 4668 raft_consensus.cc:3037] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.226399 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e8df73f9b21248cdb66ad803d9f07f48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.227064 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8df73f9b21248cdb66ad803d9f07f48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.228497 5873 leader_election.cc:288] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.229184 5105 ts_tablet_manager.cc:1398] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240503 02:12:33.229815 5105 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.234309 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ced4dfb2cf541af98d50d22d1a0f0ac from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:33.235657 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4a4f248f67ce475a8a517266b83052ac" } new_table_name: "default.Table_9ffe0b87ba144949b561f827d3d06611" modify_external_catalogs: true
I20240503 02:12:33.235888 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6bb9a205ce214c72a18394200f496a52" } new_table_name: "default.Table_c13dc37c57ae4cc383b437a5b67b4361" modify_external_catalogs: true
I20240503 02:12:33.237056 4665 raft_consensus.cc:3037] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.236784 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b5f5d5bd4a4fd6a0cfcfb8cc9e924c" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:33.240370 4808 raft_consensus.cc:3037] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.245491 5105 tablet_bootstrap.cc:654] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.253080 5851 raft_consensus.cc:482] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.253269 5851 raft_consensus.cc:504] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.254180 5851 leader_election.cc:288] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.254834 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.255081 4811 raft_consensus.cc:2454] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:33.255475 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.255587 5828 raft_consensus.cc:686] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:33.255702 4525 raft_consensus.cc:2454] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:33.255824 4608 leader_election.cc:302] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:33.256107 5851 raft_consensus.cc:2781] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.256234 5851 raft_consensus.cc:482] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.256335 5851 raft_consensus.cc:3037] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.255892 5828 consensus_queue.cc:237] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.257445 5103 tablet_bootstrap.cc:654] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:33.264369 4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 67669.5us
I20240503 02:12:33.265921 5105 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.266125 5105 ts_tablet_manager.cc:1367] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240503 02:12:33.266499 5106 tablet_bootstrap.cc:492] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.266705 5106 ts_tablet_manager.cc:1367] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.002s
I20240503 02:12:33.266957 5105 raft_consensus.cc:348] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.267199 5105 raft_consensus.cc:374] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.267287 5105 raft_consensus.cc:729] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.267535 5105 consensus_queue.cc:260] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.267912 5106 raft_consensus.cc:348] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.268321 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 468) took 5409 ms. Trace:
I20240503 02:12:33.268558 5115 rpcz_store.cc:276] 0503 02:12:27.859343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.862560 (+ 3217us) service_pool.cc:225] Handling call
0503 02:12:32.994068 (+5131508us) ts_tablet_manager.cc:1120] Deleting tablet 37a54873b48847c39a2b961ab5c3806c
0503 02:12:32.994080 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.075268 (+ 81188us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.182470 (+107202us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.268266 (+ 85796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5091900}
I20240503 02:12:33.269074 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 473) took 5358 ms. Trace:
I20240503 02:12:33.269184 5115 rpcz_store.cc:276] 0503 02:12:27.910285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.911242 (+ 957us) service_pool.cc:225] Handling call
0503 02:12:33.268884 (+5357642us) ts_tablet_manager.cc:1120] Deleting tablet 0af50eb2b1c949a0a6a71e64a894e8a5
0503 02:12:33.268894 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.269040 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5343854}
I20240503 02:12:33.269533 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 474) took 5352 ms. Trace:
I20240503 02:12:33.269629 5115 rpcz_store.cc:276] 0503 02:12:27.917026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.925078 (+ 8052us) service_pool.cc:225] Handling call
0503 02:12:33.269419 (+5344341us) ts_tablet_manager.cc:1120] Deleting tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3
0503 02:12:33.269428 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.269504 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5291106}
I20240503 02:12:33.269958 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 475) took 5334 ms. Trace:
I20240503 02:12:33.270069 4668 raft_consensus.cc:2454] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:33.270790 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 37a54873b48847c39a2b961ab5c3806c (table default.Table_c0626018603746ea91e3a2d6cfc37ef8 [id=7bfc43f0062e43c0b73e2b5fae69c922]) successfully deleted
W20240503 02:12:33.271093 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af50eb2b1c949a0a6a71e64a894e8a5 already in progress: creating tablet
W20240503 02:12:33.271323 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 already in progress: creating tablet
I20240503 02:12:33.270076 5115 rpcz_store.cc:276] 0503 02:12:27.935184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.978376 (+ 43192us) service_pool.cc:225] Handling call
0503 02:12:33.269803 (+5291427us) ts_tablet_manager.cc:1120] Deleting tablet 367de4ff0a6f4abfa2924a070f387cbd
0503 02:12:33.269812 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.269912 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5255433}
I20240503 02:12:33.271962 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 477) took 5300 ms. Trace:
I20240503 02:12:33.272068 5115 rpcz_store.cc:276] 0503 02:12:27.976770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.027376 (+ 50606us) service_pool.cc:225] Handling call
0503 02:12:33.271828 (+5244452us) ts_tablet_manager.cc:1120] Deleting tablet 1ce9d746acfe4e9aaea600293470d373
0503 02:12:33.271838 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.271924 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5244104}
I20240503 02:12:33.272405 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 476) took 5325 ms. Trace:
I20240503 02:12:33.272500 5115 rpcz_store.cc:276] 0503 02:12:27.947129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.014409 (+ 67280us) service_pool.cc:225] Handling call
0503 02:12:33.272252 (+5257843us) ts_tablet_manager.cc:1120] Deleting tablet 620204f7cb35442f87eae45f84beeee1
0503 02:12:33.272260 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.272373 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5199905}
I20240503 02:12:33.273043 4462 leader_election.cc:302] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:33.273819 5873 raft_consensus.cc:2781] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.273974 5873 raft_consensus.cc:686] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:33.274260 5873 consensus_queue.cc:237] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:33.275753 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 367de4ff0a6f4abfa2924a070f387cbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 367de4ff0a6f4abfa2924a070f387cbd already in progress: creating tablet
W20240503 02:12:33.276044 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1ce9d746acfe4e9aaea600293470d373 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce9d746acfe4e9aaea600293470d373 already in progress: creating tablet
W20240503 02:12:33.276290 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 620204f7cb35442f87eae45f84beeee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 620204f7cb35442f87eae45f84beeee1 already in progress: creating tablet
I20240503 02:12:33.272826 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 486) took 5000 ms. Trace:
I20240503 02:12:33.276635 5115 rpcz_store.cc:276] 0503 02:12:28.275036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.275128 (+ 92us) service_pool.cc:225] Handling call
0503 02:12:33.272696 (+4997568us) ts_tablet_manager.cc:1120] Deleting tablet 5a915d760ba14a31895a5dc7ec5b34fd
0503 02:12:33.272706 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.272788 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4452673}
W20240503 02:12:33.277148 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5a915d760ba14a31895a5dc7ec5b34fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a915d760ba14a31895a5dc7ec5b34fd already in progress: creating tablet
I20240503 02:12:33.277246 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d35138f04f224475af324e77aa1e0191
I20240503 02:12:33.278466 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet d35138f04f224475af324e77aa1e0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.278990 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet d35138f04f224475af324e77aa1e0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.280714 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 493) took 4841 ms. Trace:
I20240503 02:12:33.280894 5115 rpcz_store.cc:276] 0503 02:12:28.439237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.442501 (+ 3264us) service_pool.cc:225] Handling call
0503 02:12:33.277497 (+4834996us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:33.277507 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.280674 (+ 3167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4434591}
W20240503 02:12:33.282022 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 607e49cde2fd4aab9f44921d3c3921a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607e49cde2fd4aab9f44921d3c3921a5 already in progress: creating tablet
I20240503 02:12:33.284047 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 519) took 4431 ms. Trace:
I20240503 02:12:33.282941 4266 catalog_manager.cc:5484] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 1 to 2, leader changed from f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65) to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 2 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:33.284724 5115 rpcz_store.cc:276] 0503 02:12:28.853079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.159860 (+306781us) service_pool.cc:225] Handling call
0503 02:12:33.283882 (+4124022us) ts_tablet_manager.cc:1120] Deleting tablet dbfe6b2b9cab4517ae7cabec4796c9ce
0503 02:12:33.283893 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.284014 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088515}
W20240503 02:12:33.285408 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet dbfe6b2b9cab4517ae7cabec4796c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfe6b2b9cab4517ae7cabec4796c9ce already in progress: creating tablet
I20240503 02:12:33.285761 5115 tablet_replica.cc:330] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:33.286031 5115 raft_consensus.cc:2229] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:33.286296 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]). Expected version 2 got 0
I20240503 02:12:33.288081 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:33.289386 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8195767d80384a6c9a4690292f6776de
I20240503 02:12:33.289685 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 035216c9475d4b59aa1ac81b285b4d8a from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:33.289883 4271 catalog_manager.cc:6554] ReplaceTablet: tablet 8195767d80384a6c9a4690292f6776de deleted and replaced by tablet b30e440f79a64e04b7e527b52299301f
I20240503 02:12:33.290222 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3432bf1cb8f2423c8510dcf83164d243" } new_table_name: "default.Table_0bfa3f665ccb4d5cb7b522ebd26a6fc9" modify_external_catalogs: true
I20240503 02:12:33.290445 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2dd047e02b0341898a33fddece68c396"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.291241 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dd047e02b0341898a33fddece68c396 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.274267 4529 raft_consensus.cc:2454] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:33.292217 4746 leader_election.cc:302] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.292326 4808 raft_consensus.cc:2454] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:33.292902 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e37994d9ab31473f8022a677b6949bde" } new_table_name: "default.Table_d39dc930c4374b27ba6d3cd1a4593cd2" modify_external_catalogs: true
I20240503 02:12:33.293569 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0920c4cdbad14eaa91462804e2bdff73" } modify_external_catalogs: true
I20240503 02:12:33.293776 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0920c4cdbad14eaa91462804e2bdff73" } modify_external_catalogs: true
I20240503 02:12:33.294762 5115 raft_consensus.cc:2258] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.295377 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 76aef6b021da4f5aad7f523b15d484ec
I20240503 02:12:33.297567 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet d35138f04f224475af324e77aa1e0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:33.298202 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16b3d1e7ce83425db0bd56457d9f2544 consensus state has no leader
I20240503 02:12:33.302595 5828 raft_consensus.cc:2781] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.303186 5115 ts_tablet_manager.cc:1875] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.303272 5828 raft_consensus.cc:686] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:33.303552 5828 consensus_queue.cc:237] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.304225 5105 ts_tablet_manager.cc:1398] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
W20240503 02:12:33.306527 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f5506130048488caa3ee18ff7229835 consensus state has no leader
I20240503 02:12:33.268196 5106 raft_consensus.cc:374] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.308725 5106 raft_consensus.cc:729] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.309926 5106 consensus_queue.cc:260] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.313710 5992 raft_consensus.cc:482] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.313901 5992 raft_consensus.cc:504] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.314107 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.314594 5992 leader_election.cc:288] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.314654 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.314889 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.315387 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.315622 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:33.316260 4746 leader_election.cc:341] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.316524 4746 leader_election.cc:341] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.316661 4746 leader_election.cc:302] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:33.317186 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aef6b021da4f5aad7f523b15d484ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.317483 5991 raft_consensus.cc:2726] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.319067 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aef6b021da4f5aad7f523b15d484ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.319514 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aef6b021da4f5aad7f523b15d484ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.320860 4668 raft_consensus.cc:3037] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.321753 4529 raft_consensus.cc:1261] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:33.322253 5991 consensus_queue.cc:1035] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.323770 5106 ts_tablet_manager.cc:1398] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.057s user 0.002s sys 0.002s
I20240503 02:12:33.324285 5106 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.328346 5106 tablet_bootstrap.cc:654] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:33.330371 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f66e477b48246019fc2151480479a22 has no consensus state
I20240503 02:12:33.335096 5103 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.335372 5103 ts_tablet_manager.cc:1367] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.175s user 0.005s sys 0.000s
I20240503 02:12:33.336302 5103 raft_consensus.cc:348] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.336755 5103 raft_consensus.cc:374] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.336931 5103 raft_consensus.cc:729] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.337291 5103 consensus_queue.cc:260] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.337965 5103 ts_tablet_manager.cc:1398] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.338342 5926 tablet.cc:1722] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:33.338697 5103 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.341084 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c757a3a979c2489daa621cf8ea293ea8
W20240503 02:12:33.341447 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 197bca686b5e4d6cb3e6c846ed8eb584 consensus state has no leader
I20240503 02:12:33.342494 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet c757a3a979c2489daa621cf8ea293ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.342562 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet c757a3a979c2489daa621cf8ea293ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.343103 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet c757a3a979c2489daa621cf8ea293ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.346920 5105 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.356397 5103 tablet_bootstrap.cc:654] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.356663 5105 tablet_bootstrap.cc:654] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.360836 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 17afd88bf6c040ad9e8d571eccd348dc (table default.Table_8b93d7e7be90441ea960cdf3b81e4fc7 [id=5febc608abd84e7ba26288f2e155d449]). Expected version 1 got 0
I20240503 02:12:33.362524 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:33.362581 5873 raft_consensus.cc:482] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.362835 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:33.362747 5873 raft_consensus.cc:504] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.363155 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a803f06954d8a9f34e7fcf3111300 (DEFAULT_TABLE table=default.Table_018e8f0e1c0349d08ccce34247920629 [id=7ecdf02d0e9c4f3183bd10a29f958168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.364230 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.364269 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:33.365324 4462 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.365731 4462 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.365867 4462 leader_election.cc:302] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:33.366153 5868 raft_consensus.cc:2726] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.366194 6004 tablet.cc:1722] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:33.366495 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a803f06954d8a9f34e7fcf3111300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.371381 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 9327a9e5472b4eda854320c3a95f8e56 with delete_type TABLET_DATA_DELETED (replaced by tablet 3478033c4ed94268a2436a2942c27a68) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.371690 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.372126 4266 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_11cc9c0da9b14e2dab0e03b3373ad949 [id=608b3c3d30b14ddca9b21dcf3f985e57] alter complete (version 1)
I20240503 02:12:33.373631 5873 leader_election.cc:288] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.373618 5990 consensus_queue.cc:1035] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.374321 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.374603 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.378649 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.378980 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.378948 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a803f06954d8a9f34e7fcf3111300 (DEFAULT_TABLE table=default.Table_018e8f0e1c0349d08ccce34247920629 [id=7ecdf02d0e9c4f3183bd10a29f958168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.379215 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.379478 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.379760 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.379719 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a803f06954d8a9f34e7fcf3111300 (DEFAULT_TABLE table=default.Table_018e8f0e1c0349d08ccce34247920629 [id=7ecdf02d0e9c4f3183bd10a29f958168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.380049 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.381073 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 50b60601e87d4a1f9ab9b92099df5ca5 (DEFAULT_TABLE table=default.Table_269475386cb84ba993cddceff2c15394 [id=a7623d7ff3784147b431409c3b7de55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.381201 5114 ts_tablet_manager.cc:1888] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:33.381350 5114 log.cc:1196] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/79884b3e73904c708d0d48bca85351b7
I20240503 02:12:33.381806 5114 ts_tablet_manager.cc:1909] T 79884b3e73904c708d0d48bca85351b7 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:33.382875 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 50b60601e87d4a1f9ab9b92099df5ca5 (DEFAULT_TABLE table=default.Table_269475386cb84ba993cddceff2c15394 [id=a7623d7ff3784147b431409c3b7de55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.383301 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b60601e87d4a1f9ab9b92099df5ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.385545 5159 ts_tablet_manager.cc:1888] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:33.385696 5159 log.cc:1196] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/bbab4368e73b4944b9853c456a4db7bd
I20240503 02:12:33.385825 4651 tablet_service.cc:1467] Processing CreateTablet for tablet ee16f1beb6a34a85bc872dedd64472ce (DEFAULT_TABLE table=default.Table_4469969f6dd045b5834db5a53c4ff0cb [id=df7c27efcfc14936ab7194bdb4d88bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.386195 5159 ts_tablet_manager.cc:1909] T bbab4368e73b4944b9853c456a4db7bd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:33.386360 5828 raft_consensus.cc:482] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.386528 5828 raft_consensus.cc:504] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.386895 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a803f06954d8a9f34e7fcf3111300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.388351 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.388653 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.389142 5828 leader_election.cc:288] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:33.389122 4746 leader_election.cc:341] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.389364 4525 raft_consensus.cc:2454] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:33.389894 4746 leader_election.cc:302] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.389940 4506 tablet_service.cc:1467] Processing CreateTablet for tablet ee16f1beb6a34a85bc872dedd64472ce (DEFAULT_TABLE table=default.Table_4469969f6dd045b5834db5a53c4ff0cb [id=df7c27efcfc14936ab7194bdb4d88bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.390376 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee16f1beb6a34a85bc872dedd64472ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.394184 5828 raft_consensus.cc:482] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.394224 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 61f2534d4e27419e9a3272de1abc4373 (DEFAULT_TABLE table=default.Table_db309e0e41744d558fd04a5e5766718f [id=254b770a116e4a1d860d8063b33d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.394431 5828 raft_consensus.cc:504] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.395890 5828 leader_election.cc:288] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.396015 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 61f2534d4e27419e9a3272de1abc4373 (DEFAULT_TABLE table=default.Table_db309e0e41744d558fd04a5e5766718f [id=254b770a116e4a1d860d8063b33d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.396313 4786 tablet_service.cc:1467] Processing CreateTablet for tablet ee16f1beb6a34a85bc872dedd64472ce (DEFAULT_TABLE table=default.Table_4469969f6dd045b5834db5a53c4ff0cb [id=df7c27efcfc14936ab7194bdb4d88bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.396502 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f2534d4e27419e9a3272de1abc4373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.396689 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee16f1beb6a34a85bc872dedd64472ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.401350 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 61f2534d4e27419e9a3272de1abc4373 (DEFAULT_TABLE table=default.Table_db309e0e41744d558fd04a5e5766718f [id=254b770a116e4a1d860d8063b33d3611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.401789 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f2534d4e27419e9a3272de1abc4373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.404536 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c713f241c06a4a7c8106ba83bcab0c70" } modify_external_catalogs: true
I20240503 02:12:33.404781 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c713f241c06a4a7c8106ba83bcab0c70" } modify_external_catalogs: true
I20240503 02:12:33.405294 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 50b60601e87d4a1f9ab9b92099df5ca5 (DEFAULT_TABLE table=default.Table_269475386cb84ba993cddceff2c15394 [id=a7623d7ff3784147b431409c3b7de55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.405689 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b60601e87d4a1f9ab9b92099df5ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.408207 5105 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.408356 5105 ts_tablet_manager.cc:1367] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240503 02:12:33.410187 5105 raft_consensus.cc:348] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.410408 5105 raft_consensus.cc:374] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.410468 5105 raft_consensus.cc:729] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.410665 5105 consensus_queue.cc:260] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.411485 5105 ts_tablet_manager.cc:1398] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.411782 5105 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.413300 5105 tablet_bootstrap.cc:654] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.417631 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 9d900a0dc6c34076a0993210693a57a0 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.420667 4266 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_8b93d7e7be90441ea960cdf3b81e4fc7 [id=5febc608abd84e7ba26288f2e155d449] alter complete (version 1)
I20240503 02:12:33.422761 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.423264 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 09005fb01c384122a7310c1eea12d3f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 41d44fe8f6334572996665bdec60e2e5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.423552 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 617ad14927c1441b800a204aefc2e619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.423792 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.425864 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 857d019b8ab94be1b7719fec83e57e55
I20240503 02:12:33.426358 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.427345 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 857d019b8ab94be1b7719fec83e57e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.427374 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 857d019b8ab94be1b7719fec83e57e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.426995 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 857d019b8ab94be1b7719fec83e57e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.430869 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a803f06954d8a9f34e7fcf3111300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.431125 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3fb60259f2e544acae5996f00e2a3d68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.431296 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c31a7c227a7045118fb177b1b3ce2044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.431783 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fb60259f2e544acae5996f00e2a3d68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.431855 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3c2d73cae23744ad8a1b256940d3a750" } modify_external_catalogs: true
I20240503 02:12:33.431949 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fa770947e71d4e70a14a390ee36057db" } new_table_name: "default.Table_f99fb87942724549bded0fb6c29f3420" modify_external_catalogs: true
I20240503 02:12:33.432035 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3c2d73cae23744ad8a1b256940d3a750" } modify_external_catalogs: true
I20240503 02:12:33.432444 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2b00f84bf3e64c22b77142696ebcf830"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.432983 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b00f84bf3e64c22b77142696ebcf830 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:33.433964 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c31a7c227a7045118fb177b1b3ce2044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.434885 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_57b96feb46144b91851ba8aa2ba702cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.435366 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57b96feb46144b91851ba8aa2ba702cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.436405 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6742a0406a9945e2ba8eb04f293bb1c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.437093 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6742a0406a9945e2ba8eb04f293bb1c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.438206 5873 raft_consensus.cc:482] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.438364 5873 raft_consensus.cc:504] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.439052 5873 leader_election.cc:288] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.443987 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 9d900a0dc6c34076a0993210693a57a0 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.444002 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.430609 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 9d900a0dc6c34076a0993210693a57a0 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:33.448585 4462 leader_election.cc:341] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.448783 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 58942.3us
I20240503 02:12:33.452075 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.453305 5106 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.453584 5106 ts_tablet_manager.cc:1367] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.009s
I20240503 02:12:33.454448 5106 raft_consensus.cc:348] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.454890 5106 raft_consensus.cc:374] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.454985 5106 raft_consensus.cc:729] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.457158 5106 consensus_queue.cc:260] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.457365 5990 raft_consensus.cc:2781] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.457482 5990 raft_consensus.cc:482] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.457553 5990 raft_consensus.cc:3037] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:33.458760 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4225c1bc133d4e18a4a47fd58820ca10 consensus state has no leader
I20240503 02:12:33.457768 5106 ts_tablet_manager.cc:1398] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:33.459511 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 16fcfe6e2dc844a486476bf4be80d5cb
I20240503 02:12:33.461524 5990 raft_consensus.cc:504] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.461838 5115 ts_tablet_manager.cc:1888] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:33.462311 5990 leader_election.cc:288] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.464406 5106 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.466013 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 538) took 3223 ms. Trace:
I20240503 02:12:33.466141 5159 rpcz_store.cc:276] 0503 02:12:30.242438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.242547 (+ 109us) service_pool.cc:225] Handling call
0503 02:12:33.164789 (+2922242us) ts_tablet_manager.cc:1120] Deleting tablet bbab4368e73b4944b9853c456a4db7bd
0503 02:12:33.164803 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.306984 (+142181us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.385364 (+ 78380us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.465949 (+ 80585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915314}
I20240503 02:12:33.466588 5159 tablet_replica.cc:330] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:33.466773 5159 raft_consensus.cc:2229] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:33.466748 5851 raft_consensus.cc:504] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.466920 5159 raft_consensus.cc:2258] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.467545 5851 leader_election.cc:288] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.468081 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:33.468245 5159 ts_tablet_manager.cc:1875] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.468289 4808 raft_consensus.cc:3037] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.468506 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b60601e87d4a1f9ab9b92099df5ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.471068 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a3ca705e8482a8cd6a6fb17d93470" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:33.471307 4525 raft_consensus.cc:3037] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.474951 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d900a0dc6c34076a0993210693a57a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.475440 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee16f1beb6a34a85bc872dedd64472ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.476205 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f2534d4e27419e9a3272de1abc4373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.476724 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d900a0dc6c34076a0993210693a57a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.461162 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 465) took 5780 ms. Trace:
I20240503 02:12:33.478889 5114 rpcz_store.cc:276] 0503 02:12:27.680071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.680178 (+ 107us) service_pool.cc:225] Handling call
0503 02:12:33.186555 (+5506377us) ts_tablet_manager.cc:1120] Deleting tablet 79884b3e73904c708d0d48bca85351b7
0503 02:12:33.186567 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.310948 (+124381us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.381037 (+ 70089us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.461009 (+ 79972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5505767}
I20240503 02:12:33.479350 5114 tablet_replica.cc:330] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:33.479416 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.479691 5114 raft_consensus.cc:2229] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:33.479786 5115 log.cc:1196] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/4064ff59de474e0db3a578eae10c3fa6
I20240503 02:12:33.479959 5114 raft_consensus.cc:2258] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.483361 5115 ts_tablet_manager.cc:1909] T 4064ff59de474e0db3a578eae10c3fa6 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:33.483331 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:33.483412 4746 leader_election.cc:341] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.484133 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:33.486243 5114 ts_tablet_manager.cc:1875] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.487746 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:33.487975 4529 raft_consensus.cc:3037] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:33.488519 4462 leader_election.cc:341] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.488657 4462 leader_election.cc:302] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:33.489149 5868 raft_consensus.cc:2726] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:33.492097 4746 leader_election.cc:341] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.492273 4746 leader_election.cc:302] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:33.492574 5990 raft_consensus.cc:2726] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.493347 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.498602 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23526249cb074ae5b90c63d59d47e828" } modify_external_catalogs: true
I20240503 02:12:33.498842 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23526249cb074ae5b90c63d59d47e828" } modify_external_catalogs: true
I20240503 02:12:33.500736 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.501307 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 74161a00bc214f2681cb28e499d2c666 (DEFAULT_TABLE table=default.Table_7d5c9a77b56a45458afbc72f38441580 [id=155a1b2eca50425abb06077224566671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.503737 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e6175ee8a08c4522912024d62f15a984"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.504184 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_40eee5fde9014729a9372235d138bda5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.504427 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6175ee8a08c4522912024d62f15a984 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:33.504694 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40eee5fde9014729a9372235d138bda5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.506625 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c7c7ae25ece740be989d62942a8ae045" } new_table_name: "default.Table_9d93e9e41b354e29a8f41a5dfdf3a32a" modify_external_catalogs: true
I20240503 02:12:33.508066 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.508687 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.507669 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a1279c4656b24d92a8a44b6f28a0616e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.510737 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1279c4656b24d92a8a44b6f28a0616e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.511516 5868 raft_consensus.cc:482] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.511718 5868 raft_consensus.cc:504] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.512419 5868 leader_election.cc:288] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.512517 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74161a00bc214f2681cb28e499d2c666. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:33.512910 4462 consensus_peers.cc:485] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:33.514117 5105 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.514262 5105 ts_tablet_manager.cc:1367] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.103s user 0.013s sys 0.000s
I20240503 02:12:33.514766 5106 tablet_bootstrap.cc:654] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.514961 5105 raft_consensus.cc:348] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.515151 5105 raft_consensus.cc:374] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.515202 5105 raft_consensus.cc:729] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.515421 5105 consensus_queue.cc:260] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.517691 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 74161a00bc214f2681cb28e499d2c666 (DEFAULT_TABLE table=default.Table_7d5c9a77b56a45458afbc72f38441580 [id=155a1b2eca50425abb06077224566671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.518114 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74161a00bc214f2681cb28e499d2c666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.518072 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 74161a00bc214f2681cb28e499d2c666 (DEFAULT_TABLE table=default.Table_7d5c9a77b56a45458afbc72f38441580 [id=155a1b2eca50425abb06077224566671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.519034 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 6f66e477b48246019fc2151480479a22 (DEFAULT_TABLE table=default.Table_0bfa3f665ccb4d5cb7b522ebd26a6fc9 [id=3c226023bc2c4f158dfbe027d38f3fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.519455 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f66e477b48246019fc2151480479a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.521807 5105 ts_tablet_manager.cc:1398] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:12:33.522125 5105 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.523626 5105 tablet_bootstrap.cc:654] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.524158 4670 raft_consensus.cc:3037] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.524443 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 6f66e477b48246019fc2151480479a22 (DEFAULT_TABLE table=default.Table_0bfa3f665ccb4d5cb7b522ebd26a6fc9 [id=3c226023bc2c4f158dfbe027d38f3fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.523991 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 6f66e477b48246019fc2151480479a22 (DEFAULT_TABLE table=default.Table_0bfa3f665ccb4d5cb7b522ebd26a6fc9 [id=3c226023bc2c4f158dfbe027d38f3fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.528613 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d900a0dc6c34076a0993210693a57a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.529685 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:33.530301 4462 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.531134 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f66e477b48246019fc2151480479a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.538674 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 79884b3e73904c708d0d48bca85351b7 (table default.Table_b839b2ab5a8a4a399e63762bec827155 [id=156d16ce2a4246cd81968e098a9227dc]) successfully deleted
I20240503 02:12:33.539294 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet bbab4368e73b4944b9853c456a4db7bd (table default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]) successfully deleted
I20240503 02:12:33.539657 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_01fcdbe2fef14f7284c1f68c4f72e92c" } new_table_name: "default.Table_31e8871df31d4f888262f6952dd78987" modify_external_catalogs: true
I20240503 02:12:33.540982 5105 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.541123 5105 ts_tablet_manager.cc:1367] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
W20240503 02:12:33.541667 4746 leader_election.cc:341] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.541878 5105 raft_consensus.cc:348] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:33.542371 5105 raft_consensus.cc:374] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.542434 5105 raft_consensus.cc:729] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.528263 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7566b876f57547599a98c4f55fbfc427" } modify_external_catalogs: true
I20240503 02:12:33.542634 5105 consensus_queue.cc:260] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:33.542905 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7566b876f57547599a98c4f55fbfc427" } modify_external_catalogs: true
I20240503 02:12:33.529276 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2dd047e02b0341898a33fddece68c396" } new_table_name: "default.Table_04015aedd8a14fecbad557d674b75f31" modify_external_catalogs: true
I20240503 02:12:33.543556 5105 ts_tablet_manager.cc:1398] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.543833 5105 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.546278 5105 tablet_bootstrap.cc:654] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.546746 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1aea91e895ce4610be1ebf9ee910958b" } new_table_name: "default.Table_3e7dad4cf88048efa489e61c2595ad65" modify_external_catalogs: true
I20240503 02:12:33.546803 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fcfe6e2dc844a486476bf4be80d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.549428 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.549626 4811 raft_consensus.cc:2454] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:33.552313 4462 leader_election.cc:302] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.552805 5868 consensus_queue.cc:1035] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.553385 5868 raft_consensus.cc:2781] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.553489 5868 raft_consensus.cc:482] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.553598 5868 raft_consensus.cc:3037] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.555503 5103 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.555759 5103 ts_tablet_manager.cc:1367] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.217s user 0.008s sys 0.004s
I20240503 02:12:33.556236 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet dee34be89bda450296b916163bb10c46
I20240503 02:12:33.556782 4668 raft_consensus.cc:1261] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:33.558663 4525 raft_consensus.cc:2454] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:33.559602 4608 leader_election.cc:302] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:33.562310 4808 raft_consensus.cc:2454] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:33.563339 5103 raft_consensus.cc:348] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:33.563799 5103 raft_consensus.cc:374] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.563887 5103 raft_consensus.cc:729] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.564266 5103 consensus_queue.cc:260] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:33.564853 5103 ts_tablet_manager.cc:1398] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s
I20240503 02:12:33.565110 5851 raft_consensus.cc:2781] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.565270 5851 raft_consensus.cc:686] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:33.565707 5851 consensus_queue.cc:237] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.566408 5873 raft_consensus.cc:482] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.566563 5873 raft_consensus.cc:504] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.567266 5873 leader_election.cc:288] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.568224 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:33.569088 4462 leader_election.cc:341] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.571820 5103 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.574896 5103 tablet_bootstrap.cc:654] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.577808 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 522) took 4724 ms. Trace:
I20240503 02:12:33.577951 5115 rpcz_store.cc:276] 0503 02:12:28.853428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.244017 (+390589us) service_pool.cc:225] Handling call
0503 02:12:33.285692 (+4041675us) ts_tablet_manager.cc:1120] Deleting tablet 4064ff59de474e0db3a578eae10c3fa6
0503 02:12:33.285703 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.364775 (+ 79072us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.461432 (+ 96657us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.577732 (+116300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3084394}
I20240503 02:12:33.580998 5103 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.582204 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e635f283288406aa16b3941be64e777 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:33.582352 5103 ts_tablet_manager.cc:1367] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 02:12:33.586733 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 523) took 4724 ms. Trace:
I20240503 02:12:33.588119 5103 raft_consensus.cc:348] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.588691 5990 consensus_queue.cc:1035] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.593272 5103 raft_consensus.cc:374] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.593703 5103 raft_consensus.cc:729] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.594329 5103 consensus_queue.cc:260] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.595548 5103 ts_tablet_manager.cc:1398] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:33.596593 5103 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.597179 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet dee34be89bda450296b916163bb10c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.597307 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet dee34be89bda450296b916163bb10c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.600565 5103 tablet_bootstrap.cc:654] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.601351 5115 rpcz_store.cc:276] 0503 02:12:28.853516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.246258 (+392742us) service_pool.cc:225] Handling call
0503 02:12:33.578239 (+4331981us) ts_tablet_manager.cc:1120] Deleting tablet 0cc2ebfb11aa4840adcc35da73be75a2
0503 02:12:33.578254 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.578381 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3376117}
I20240503 02:12:33.606699 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74161a00bc214f2681cb28e499d2c666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.607398 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f66e477b48246019fc2151480479a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.607574 5873 raft_consensus.cc:482] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.607556 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.607766 5873 raft_consensus.cc:504] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.608701 5873 leader_election.cc:288] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.621248 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet dee34be89bda450296b916163bb10c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:33.621497 5115 tablet_replica.cc:330] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:33.621812 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:33.622221 4462 leader_election.cc:341] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.622442 4462 leader_election.cc:302] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:33.583979 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 890fb92ca2314f2e81e1ffb0bcddb0b1 consensus state has no leader
I20240503 02:12:33.625229 5873 raft_consensus.cc:2726] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:33.584501 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_00deed6cfb98406d84258657464b8f93" } modify_external_catalogs: true
I20240503 02:12:33.585865 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f235f17c19bd4117b7a88a40edb5e3ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.626410 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_00deed6cfb98406d84258657464b8f93" } modify_external_catalogs: true
W20240503 02:12:33.626901 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f235f17c19bd4117b7a88a40edb5e3ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.595939 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_77eae5a54c4c440a9928ae87553755be [id=5108d90cc915403ba879db13f6a9f1be]) successfully deleted
W20240503 02:12:33.634018 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0cc2ebfb11aa4840adcc35da73be75a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc2ebfb11aa4840adcc35da73be75a2 already in progress: creating tablet
I20240503 02:12:33.634405 5115 raft_consensus.cc:2229] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:33.634837 5115 raft_consensus.cc:2258] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.636787 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:33.637727 4462 leader_election.cc:341] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.641053 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9c1a5d3ddc24b44b921648bd94ad827 consensus state has no leader
I20240503 02:12:33.641638 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d6e976e60b3e445492636b06856344c3" } modify_external_catalogs: true
I20240503 02:12:33.641906 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d6e976e60b3e445492636b06856344c3" } modify_external_catalogs: true
I20240503 02:12:33.642024 5115 ts_tablet_manager.cc:1875] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:33.643951 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 consensus state has no leader
W20240503 02:12:33.645558 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c084ab3fedb46d4992f5af2ceede06b has no consensus state
I20240503 02:12:33.649874 6020 tablet.cc:1722] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:33.650871 4668 raft_consensus.cc:3037] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.652086 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_53cb05a18acb413295d4cfe37fb3b76d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.652524 4529 raft_consensus.cc:2454] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
W20240503 02:12:33.652977 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53cb05a18acb413295d4cfe37fb3b76d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.654182 5873 consensus_queue.cc:1035] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.655392 5868 raft_consensus.cc:504] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.656563 5868 leader_election.cc:288] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.657327 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:33.657913 4462 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:33.660985 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4225c1bc133d4e18a4a47fd58820ca10 (table default.Table_f99fb87942724549bded0fb6c29f3420 [id=5cbfe61d82174908b3b00ff5d287eb8f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4225c1bc133d4e18a4a47fd58820ca10 consensus state has no leader
I20240503 02:12:33.662539 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_95aad9f7fca14d9fb0d98d25071f9f08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.663365 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95aad9f7fca14d9fb0d98d25071f9f08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.676501 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_dd49ed261dd44569888a7e6f0dab8c66" } new_table_name: "default.Table_9cafcae38e354ce9af1528fa96cdc481" modify_external_catalogs: true
W20240503 02:12:33.679075 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 47291.5us
I20240503 02:12:33.686184 5114 ts_tablet_manager.cc:1888] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:33.686493 5114 log.cc:1196] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/34eb69d5f4bd427a8e430a00fb076f83
I20240503 02:12:33.686654 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.686870 4810 raft_consensus.cc:2454] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:33.686946 5990 raft_consensus.cc:482] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.686988 4746 leader_election.cc:302] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.687043 5990 raft_consensus.cc:504] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.687147 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:33.687249 5828 raft_consensus.cc:2781] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.687294 4807 raft_consensus.cc:3037] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.687355 5828 raft_consensus.cc:686] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:33.687589 5828 consensus_queue.cc:237] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.687711 5990 leader_election.cc:288] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.689667 5114 ts_tablet_manager.cc:1909] T 34eb69d5f4bd427a8e430a00fb076f83 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:33.693691 5159 ts_tablet_manager.cc:1888] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:33.693950 5159 log.cc:1196] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/09005fb01c384122a7310c1eea12d3f0
I20240503 02:12:33.694398 5159 ts_tablet_manager.cc:1909] T 09005fb01c384122a7310c1eea12d3f0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:33.694918 5873 raft_consensus.cc:482] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.695140 5873 raft_consensus.cc:504] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.695852 5873 leader_election.cc:288] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.697641 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.702129 5999 raft_consensus.cc:482] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.702286 5999 raft_consensus.cc:504] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.702980 5999 leader_election.cc:288] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.703563 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.703792 4810 raft_consensus.cc:2454] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:33.703742 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.704464 4608 leader_election.cc:302] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:33.704824 5999 raft_consensus.cc:2781] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.704932 5999 raft_consensus.cc:482] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.705022 5999 raft_consensus.cc:3037] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:33.705206 4608 leader_election.cc:341] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.707376 5106 tablet_bootstrap.cc:492] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.707963 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:33.709813 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3a2b0f0f386845c18712a0ddb23d9cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.710791 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a2b0f0f386845c18712a0ddb23d9cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.713709 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:33.716696 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0134301a980f41e5847132881085032e" } modify_external_catalogs: true
I20240503 02:12:33.716971 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0134301a980f41e5847132881085032e" } modify_external_catalogs: true
I20240503 02:12:33.717502 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.718549 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 6e045884d4a24772940cf711a3b84f07 (DEFAULT_TABLE table=default.Table_c31a7c227a7045118fb177b1b3ce2044 [id=c7702061be80496e8fcf5d5dd6710d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.718667 5106 ts_tablet_manager.cc:1367] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.259s user 0.017s sys 0.026s
I20240503 02:12:33.719921 5106 raft_consensus.cc:348] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.720273 5106 raft_consensus.cc:374] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.720386 5106 raft_consensus.cc:729] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.720707 5106 consensus_queue.cc:260] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.721077 6001 raft_consensus.cc:482] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.721217 6001 raft_consensus.cc:504] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.721911 6001 leader_election.cc:288] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.722713 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 6e045884d4a24772940cf711a3b84f07 (DEFAULT_TABLE table=default.Table_c31a7c227a7045118fb177b1b3ce2044 [id=c7702061be80496e8fcf5d5dd6710d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.723285 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e045884d4a24772940cf711a3b84f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.723285 5991 raft_consensus.cc:482] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.723378 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.723459 5991 raft_consensus.cc:504] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.723605 4810 raft_consensus.cc:2454] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:33.724144 5991 leader_election.cc:288] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.725561 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3e904ed115416f9877e34d2986f75d (DEFAULT_TABLE table=default.Table_8eefedb5829e47eaba1cf863483cd06d [id=e0b52e4a8eab46f9b341c07a4fcdc014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.724990 4608 leader_election.cc:302] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:33.726275 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 6e045884d4a24772940cf711a3b84f07 (DEFAULT_TABLE table=default.Table_c31a7c227a7045118fb177b1b3ce2044 [id=c7702061be80496e8fcf5d5dd6710d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.726245 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.726523 4529 raft_consensus.cc:2454] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:33.726994 6001 raft_consensus.cc:2781] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.727098 6001 raft_consensus.cc:482] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.727169 6001 raft_consensus.cc:3037] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.728363 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3e904ed115416f9877e34d2986f75d (DEFAULT_TABLE table=default.Table_8eefedb5829e47eaba1cf863483cd06d [id=e0b52e4a8eab46f9b341c07a4fcdc014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.728890 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3e904ed115416f9877e34d2986f75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.729753 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e045884d4a24772940cf711a3b84f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.733868 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3e904ed115416f9877e34d2986f75d (DEFAULT_TABLE table=default.Table_8eefedb5829e47eaba1cf863483cd06d [id=e0b52e4a8eab46f9b341c07a4fcdc014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.734334 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3e904ed115416f9877e34d2986f75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.739751 5851 raft_consensus.cc:482] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.740007 5851 raft_consensus.cc:504] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.740226 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 (DEFAULT_TABLE table=default.Table_885ac00da36e484f81d9bdba48c6e155 [id=9f292257da7f464ea5ff915478dadec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.741549 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 (DEFAULT_TABLE table=default.Table_885ac00da36e484f81d9bdba48c6e155 [id=9f292257da7f464ea5ff915478dadec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.742071 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:33.744459 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 57488.5us
I20240503 02:12:33.745575 5851 leader_election.cc:288] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.746243 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9327a9e5472b4eda854320c3a95f8e56" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.746584 4529 raft_consensus.cc:2379] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:33.745924 5106 ts_tablet_manager.cc:1398] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20240503 02:12:33.747098 5106 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:33.747866 4462 consensus_peers.cc:485] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbab4368e73b4944b9853c456a4db7bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:33.751433 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 464) took 6071 ms. Trace:
I20240503 02:12:33.751853 4807 raft_consensus.cc:2454] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:33.752503 4462 leader_election.cc:302] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:33.753177 5873 raft_consensus.cc:2781] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.753284 5873 raft_consensus.cc:482] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.753392 5873 raft_consensus.cc:3037] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.751597 5114 rpcz_store.cc:276] 0503 02:12:27.679975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.680211 (+ 236us) service_pool.cc:225] Handling call
0503 02:12:33.479235 (+5799024us) ts_tablet_manager.cc:1120] Deleting tablet 34eb69d5f4bd427a8e430a00fb076f83
0503 02:12:33.479247 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.606094 (+126847us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.685977 (+ 79883us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.751386 (+ 65409us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":5797294}
I20240503 02:12:33.757208 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.757614 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 467) took 6072 ms. Trace:
I20240503 02:12:33.757741 5114 rpcz_store.cc:276] 0503 02:12:27.684999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.685968 (+ 969us) service_pool.cc:225] Handling call
0503 02:12:33.757444 (+6071476us) ts_tablet_manager.cc:1120] Deleting tablet acdefb33fe6e4a9893dd66f6252395d7
0503 02:12:33.757456 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.757572 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5614182}
I20240503 02:12:33.758064 5114 tablet_replica.cc:330] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:33.758332 5114 raft_consensus.cc:2229] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:33.758689 5114 raft_consensus.cc:2258] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.759140 5106 tablet_bootstrap.cc:654] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.759719 5114 ts_tablet_manager.cc:1875] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.759920 6013 consensus_queue.cc:1035] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.761938 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 539) took 3517 ms. Trace:
I20240503 02:12:33.762046 5159 rpcz_store.cc:276] 0503 02:12:30.244087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.244172 (+ 85us) service_pool.cc:225] Handling call
0503 02:12:33.466535 (+3222363us) ts_tablet_manager.cc:1120] Deleting tablet 09005fb01c384122a7310c1eea12d3f0
0503 02:12:33.466545 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.590661 (+124116us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.693571 (+102910us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.761910 (+ 68339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216958}
I20240503 02:12:33.769328 5159 tablet_replica.cc:330] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:33.769594 5159 raft_consensus.cc:2229] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:33.769918 5159 raft_consensus.cc:2258] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.770535 5159 ts_tablet_manager.cc:1875] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:33.777751 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 (DEFAULT_TABLE table=default.Table_885ac00da36e484f81d9bdba48c6e155 [id=9f292257da7f464ea5ff915478dadec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.778264 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.778154 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 7447242714c844e685faca5aed71a7f7 (DEFAULT_TABLE table=default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b [id=ccdae6dae1f049bf806ef9018cb6cace]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.779230 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 7447242714c844e685faca5aed71a7f7 (DEFAULT_TABLE table=default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b [id=ccdae6dae1f049bf806ef9018cb6cace]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.780262 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 7447242714c844e685faca5aed71a7f7 (DEFAULT_TABLE table=default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b [id=ccdae6dae1f049bf806ef9018cb6cace]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.780671 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447242714c844e685faca5aed71a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.783288 5106 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.783459 5106 ts_tablet_manager.cc:1367] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.005s
I20240503 02:12:33.785326 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.789184 4636 tablet_service.cc:1467] Processing CreateTablet for tablet ce7b279d0df9438f9e90b01c4e619c7d (DEFAULT_TABLE table=default.Table_e8df73f9b21248cdb66ad803d9f07f48 [id=17d498587e4d4d5ab2b3790a9bd4bfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.789554 5106 raft_consensus.cc:348] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.790941 5106 raft_consensus.cc:374] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.792081 5106 raft_consensus.cc:729] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.792975 4495 tablet_service.cc:1467] Processing CreateTablet for tablet ce7b279d0df9438f9e90b01c4e619c7d (DEFAULT_TABLE table=default.Table_e8df73f9b21248cdb66ad803d9f07f48 [id=17d498587e4d4d5ab2b3790a9bd4bfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.793391 4645 tablet_service.cc:1467] Processing CreateTablet for tablet cef84b4e6ba24150905994d45ffd0b52 (DEFAULT_TABLE table=default.Table_930427a07e994d4f8ad39f8cd1881e2a [id=277825cb543a4229a5f5e9506f8c5dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.793004 5106 consensus_queue.cc:260] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.794636 4506 tablet_service.cc:1467] Processing CreateTablet for tablet cef84b4e6ba24150905994d45ffd0b52 (DEFAULT_TABLE table=default.Table_930427a07e994d4f8ad39f8cd1881e2a [id=277825cb543a4229a5f5e9506f8c5dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.795553 4639 tablet_service.cc:1467] Processing CreateTablet for tablet e489b424b2d546e2aa993163f13dcbd9 (DEFAULT_TABLE table=default.Table_57b96feb46144b91851ba8aa2ba702cf [id=32b09031d84b4d46880ed8a1ed057403]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.796856 4505 tablet_service.cc:1467] Processing CreateTablet for tablet e489b424b2d546e2aa993163f13dcbd9 (DEFAULT_TABLE table=default.Table_57b96feb46144b91851ba8aa2ba702cf [id=32b09031d84b4d46880ed8a1ed057403]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.795686 5106 ts_tablet_manager.cc:1398] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:12:33.798545 5106 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.799901 4791 tablet_service.cc:1467] Processing CreateTablet for tablet cef84b4e6ba24150905994d45ffd0b52 (DEFAULT_TABLE table=default.Table_930427a07e994d4f8ad39f8cd1881e2a [id=277825cb543a4229a5f5e9506f8c5dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.800343 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef84b4e6ba24150905994d45ffd0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.800443 5106 tablet_bootstrap.cc:654] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.800577 5105 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:33.800693 5105 ts_tablet_manager.cc:1367] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.257s user 0.038s sys 0.017s
I20240503 02:12:33.801440 5105 raft_consensus.cc:348] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.801612 5105 raft_consensus.cc:374] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.801656 5105 raft_consensus.cc:729] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.801827 5105 consensus_queue.cc:260] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.802281 5105 ts_tablet_manager.cc:1398] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.802577 5105 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:33.799901 4792 tablet_service.cc:1467] Processing CreateTablet for tablet ce7b279d0df9438f9e90b01c4e619c7d (DEFAULT_TABLE table=default.Table_e8df73f9b21248cdb66ad803d9f07f48 [id=17d498587e4d4d5ab2b3790a9bd4bfc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.803683 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7b279d0df9438f9e90b01c4e619c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.803967 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:33.804116 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:33.804275 4529 raft_consensus.cc:2454] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:33.812460 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 3c421694ce1e424da7953d061a6bca3e (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.814145 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 3c421694ce1e424da7953d061a6bca3e (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.815501 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc (DEFAULT_TABLE table=default.Table_1a5f115728484ec18350f73f8ffa672b [id=69b80b16f0bb411dac6dd8b332261b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.816991 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc (DEFAULT_TABLE table=default.Table_1a5f115728484ec18350f73f8ffa672b [id=69b80b16f0bb411dac6dd8b332261b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.818383 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 7982ba6234b844f682cb23c0bfc4068c (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.821704 5103 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:33.821768 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 7982ba6234b844f682cb23c0bfc4068c (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.821902 5103 ts_tablet_manager.cc:1367] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.225s user 0.003s sys 0.031s
I20240503 02:12:33.822670 5103 raft_consensus.cc:348] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.822939 5103 raft_consensus.cc:374] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.823024 5103 raft_consensus.cc:729] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.823271 5103 consensus_queue.cc:260] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.824594 5103 ts_tablet_manager.cc:1398] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:33.824963 5103 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:33.826855 5103 tablet_bootstrap.cc:654] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.828358 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 75aa4b8ef74e4af18aa4390d593d665d (DEFAULT_TABLE table=default.Table_2b00f84bf3e64c22b77142696ebcf830 [id=c321ac08d5a84b038c385f1145b100dc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:33.829277 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d919b9262ac48cbb8b3f893918b5590 consensus state has no leader
I20240503 02:12:33.829950 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 75aa4b8ef74e4af18aa4390d593d665d (DEFAULT_TABLE table=default.Table_2b00f84bf3e64c22b77142696ebcf830 [id=c321ac08d5a84b038c385f1145b100dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.831073 4643 tablet_service.cc:1467] Processing CreateTablet for tablet d9c8547443694ce4b4b8018f190ac49f (DEFAULT_TABLE table=default.Table_bb24fceb8c3e46a89b5f2be967010467 [id=f467d0e6293743fb94648f21c01ef8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.834578 4498 tablet_service.cc:1467] Processing CreateTablet for tablet d9c8547443694ce4b4b8018f190ac49f (DEFAULT_TABLE table=default.Table_bb24fceb8c3e46a89b5f2be967010467 [id=f467d0e6293743fb94648f21c01ef8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.835041 4648 tablet_service.cc:1467] Processing CreateTablet for tablet d224bded18ed4e64ae52d01c91cacc8f (DEFAULT_TABLE table=default.Table_3fb60259f2e544acae5996f00e2a3d68 [id=7afc61f9fecf42dcabd8b9057b7b661f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.835358 4501 tablet_service.cc:1467] Processing CreateTablet for tablet d224bded18ed4e64ae52d01c91cacc8f (DEFAULT_TABLE table=default.Table_3fb60259f2e544acae5996f00e2a3d68 [id=7afc61f9fecf42dcabd8b9057b7b661f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.836289 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa5be5a4f64c189ef619bc38e21eae (DEFAULT_TABLE table=default.Table_6742a0406a9945e2ba8eb04f293bb1c9 [id=a95b6d3cb7fd4a86931abd20d16664fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.836521 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa5be5a4f64c189ef619bc38e21eae (DEFAULT_TABLE table=default.Table_6742a0406a9945e2ba8eb04f293bb1c9 [id=a95b6d3cb7fd4a86931abd20d16664fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.837050 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4a15903f49d74e708bd42ab681cb77a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.837462 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 1c084ab3fedb46d4992f5af2ceede06b (DEFAULT_TABLE table=default.Table_04015aedd8a14fecbad557d674b75f31 [id=b655253e9eea42ac951931e69dd2c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.838456 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 1c084ab3fedb46d4992f5af2ceede06b (DEFAULT_TABLE table=default.Table_04015aedd8a14fecbad557d674b75f31 [id=b655253e9eea42ac951931e69dd2c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.838523 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_85851cb314a247e4a318be19d0c3b97e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:33.839830 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f (DEFAULT_TABLE table=default.Table_a6d78e7a628d4b7b939e74d32d5ce601 [id=bed05ac07b354f169c9cc84a553f1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.840806 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f (DEFAULT_TABLE table=default.Table_a6d78e7a628d4b7b939e74d32d5ce601 [id=bed05ac07b354f169c9cc84a553f1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.841356 4507 tablet_service.cc:1467] Processing CreateTablet for tablet b30e440f79a64e04b7e527b52299301f (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:33.841441 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a15903f49d74e708bd42ab681cb77a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:33.842643 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85851cb314a247e4a318be19d0c3b97e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.842840 4647 tablet_service.cc:1467] Processing CreateTablet for tablet b30e440f79a64e04b7e527b52299301f (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.843575 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 09005fb01c384122a7310c1eea12d3f0 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:33.844074 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 34eb69d5f4bd427a8e430a00fb076f83 (table default.Table_a6d935ff1f554a62a312091e6252d15a [id=7118c326733f4c6695909797bd035f9e]) successfully deleted
W20240503 02:12:33.844342 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet acdefb33fe6e4a9893dd66f6252395d7 because the tablet was not found. No further retry: Not found: Tablet not found: acdefb33fe6e4a9893dd66f6252395d7
I20240503 02:12:33.842900 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:33.845018 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:33.849208 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:33.851967 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01" } modify_external_catalogs: true
I20240503 02:12:33.852268 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f5ab13aaabaf44d4bf3dabf77fb55a01" } modify_external_catalogs: true
I20240503 02:12:33.853427 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7b279d0df9438f9e90b01c4e619c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.853946 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef84b4e6ba24150905994d45ffd0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.853824 5999 raft_consensus.cc:504] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.854450 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e489b424b2d546e2aa993163f13dcbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.854676 5999 leader_election.cc:288] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.854878 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c8547443694ce4b4b8018f190ac49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.854966 4668 raft_consensus.cc:2379] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:33.855396 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.855926 5873 raft_consensus.cc:504] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.856781 5873 leader_election.cc:288] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:33.857678 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:33.857909 4668 raft_consensus.cc:3037] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.858901 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c8547443694ce4b4b8018f190ac49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.859103 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d224bded18ed4e64ae52d01c91cacc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.859254 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447242714c844e685faca5aed71a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.859786 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c421694ce1e424da7953d061a6bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.860329 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.860759 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7982ba6234b844f682cb23c0bfc4068c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.861047 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa5be5a4f64c189ef619bc38e21eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.861068 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aa4b8ef74e4af18aa4390d593d665d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.861492 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d224bded18ed4e64ae52d01c91cacc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.861925 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa5be5a4f64c189ef619bc38e21eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.862423 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c084ab3fedb46d4992f5af2ceede06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.862864 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e440f79a64e04b7e527b52299301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.865836 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e045884d4a24772940cf711a3b84f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.866346 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3e904ed115416f9877e34d2986f75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.866727 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.867058 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447242714c844e685faca5aed71a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.867413 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7b279d0df9438f9e90b01c4e619c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.867831 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef84b4e6ba24150905994d45ffd0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.868219 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e489b424b2d546e2aa993163f13dcbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.868606 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c421694ce1e424da7953d061a6bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.878756 5991 raft_consensus.cc:482] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.881147 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.884255 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7982ba6234b844f682cb23c0bfc4068c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.885586 5106 tablet_bootstrap.cc:492] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.885833 5106 ts_tablet_manager.cc:1367] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.011s
I20240503 02:12:33.886626 5106 raft_consensus.cc:348] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.886808 5106 raft_consensus.cc:374] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.886848 5106 raft_consensus.cc:729] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.887085 5106 consensus_queue.cc:260] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.887583 5106 ts_tablet_manager.cc:1398] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:33.887759 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.891273 5106 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.891585 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aa4b8ef74e4af18aa4390d593d665d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.893054 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e440f79a64e04b7e527b52299301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.893417 5106 tablet_bootstrap.cc:654] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.902433 5106 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.902601 5106 ts_tablet_manager.cc:1367] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 02:12:33.905241 5106 raft_consensus.cc:348] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.905560 5106 raft_consensus.cc:374] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.905625 5106 raft_consensus.cc:729] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.906025 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c084ab3fedb46d4992f5af2ceede06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.905896 5106 consensus_queue.cc:260] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.906639 5106 ts_tablet_manager.cc:1398] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:33.907230 5106 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.914505 5106 tablet_bootstrap.cc:654] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.915879 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:33.916152 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc (DEFAULT_TABLE table=default.Table_1a5f115728484ec18350f73f8ffa672b [id=69b80b16f0bb411dac6dd8b332261b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.916635 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 3c421694ce1e424da7953d061a6bca3e (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.918200 4786 tablet_service.cc:1467] Processing CreateTablet for tablet e489b424b2d546e2aa993163f13dcbd9 (DEFAULT_TABLE table=default.Table_57b96feb46144b91851ba8aa2ba702cf [id=32b09031d84b4d46880ed8a1ed057403]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.919302 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 7982ba6234b844f682cb23c0bfc4068c (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.919737 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7982ba6234b844f682cb23c0bfc4068c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.920604 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e489b424b2d546e2aa993163f13dcbd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:33.921227 4462 leader_election.cc:341] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.922276 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c421694ce1e424da7953d061a6bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.922757 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.923828 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23be1160c454c139128aff5a960d183" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:33.925978 4807 raft_consensus.cc:3037] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.925870 5991 raft_consensus.cc:504] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.926391 5106 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.926529 5106 ts_tablet_manager.cc:1367] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 02:12:33.927263 5106 raft_consensus.cc:348] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.927446 5106 raft_consensus.cc:374] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.927490 5106 raft_consensus.cc:729] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.927667 5106 consensus_queue.cc:260] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.928203 5106 ts_tablet_manager.cc:1398] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:33.928471 5106 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.926676 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 75aa4b8ef74e4af18aa4390d593d665d (DEFAULT_TABLE table=default.Table_2b00f84bf3e64c22b77142696ebcf830 [id=c321ac08d5a84b038c385f1145b100dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.929782 5873 raft_consensus.cc:482] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.930465 4787 tablet_service.cc:1467] Processing CreateTablet for tablet d9c8547443694ce4b4b8018f190ac49f (DEFAULT_TABLE table=default.Table_bb24fceb8c3e46a89b5f2be967010467 [id=f467d0e6293743fb94648f21c01ef8bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:33.931686 4746 leader_election.cc:341] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:33.931870 4746 leader_election.cc:302] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:33.932770 5990 raft_consensus.cc:2781] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.932852 5990 raft_consensus.cc:482] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.932957 5990 raft_consensus.cc:3037] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:33.934861 4608 leader_election.cc:341] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 9327a9e5472b4eda854320c3a95f8e56
I20240503 02:12:33.934999 4608 leader_election.cc:302] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:33.936817 5999 raft_consensus.cc:3037] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.937593 6013 raft_consensus.cc:482] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.938406 5873 raft_consensus.cc:504] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.939502 5106 tablet_bootstrap.cc:654] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.939888 4780 tablet_service.cc:1467] Processing CreateTablet for tablet b30e440f79a64e04b7e527b52299301f (DEFAULT_TABLE table=default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.940131 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:33.940308 4810 raft_consensus.cc:3037] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.940558 5991 leader_election.cc:288] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:33.943799 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 1c084ab3fedb46d4992f5af2ceede06b (DEFAULT_TABLE table=default.Table_04015aedd8a14fecbad557d674b75f31 [id=b655253e9eea42ac951931e69dd2c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.944077 5106 tablet_bootstrap.cc:492] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.944092 4793 tablet_service.cc:1467] Processing CreateTablet for tablet d224bded18ed4e64ae52d01c91cacc8f (DEFAULT_TABLE table=default.Table_3fb60259f2e544acae5996f00e2a3d68 [id=7afc61f9fecf42dcabd8b9057b7b661f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.944226 5106 ts_tablet_manager.cc:1367] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20240503 02:12:33.944983 5106 raft_consensus.cc:348] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.945178 5106 raft_consensus.cc:374] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.945241 5106 raft_consensus.cc:729] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.945343 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f (DEFAULT_TABLE table=default.Table_a6d78e7a628d4b7b939e74d32d5ce601 [id=bed05ac07b354f169c9cc84a553f1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.948112 5106 consensus_queue.cc:260] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.948366 5105 tablet_bootstrap.cc:654] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.948683 5106 ts_tablet_manager.cc:1398] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:33.948683 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 77fa5be5a4f64c189ef619bc38e21eae (DEFAULT_TABLE table=default.Table_6742a0406a9945e2ba8eb04f293bb1c9 [id=a95b6d3cb7fd4a86931abd20d16664fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.949007 5106 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.952440 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c8547443694ce4b4b8018f190ac49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.952646 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aa4b8ef74e4af18aa4390d593d665d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.953197 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d224bded18ed4e64ae52d01c91cacc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.957159 5106 tablet_bootstrap.cc:654] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.957356 4668 raft_consensus.cc:2454] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:33.957957 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.958230 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fa5be5a4f64c189ef619bc38e21eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.958475 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e440f79a64e04b7e527b52299301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.959034 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c084ab3fedb46d4992f5af2ceede06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.957158 4807 raft_consensus.cc:2454] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:33.964821 5106 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:33.964996 5106 ts_tablet_manager.cc:1367] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 02:12:33.965736 5106 raft_consensus.cc:348] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.965932 5106 raft_consensus.cc:374] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.965976 5106 raft_consensus.cc:729] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.974210 5106 consensus_queue.cc:260] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:33.974821 5106 ts_tablet_manager.cc:1398] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:12:33.975109 5106 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:33.978579 5106 tablet_bootstrap.cc:654] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.978993 5115 ts_tablet_manager.cc:1888] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:33.981884 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94031465ce7407dab0f90ad8809ac21" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.000288 5106 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.016508 5106 ts_tablet_manager.cc:1367] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20240503 02:12:34.018558 5106 raft_consensus.cc:348] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.018905 5106 raft_consensus.cc:374] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.018988 5106 raft_consensus.cc:729] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.019381 5106 consensus_queue.cc:260] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.020239 5106 ts_tablet_manager.cc:1398] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:34.020468 5115 log.cc:1196] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/dedead5a2bf347f6991cce8e9b4db708
I20240503 02:12:34.021152 5115 ts_tablet_manager.cc:1909] T dedead5a2bf347f6991cce8e9b4db708 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:34.042416 5828 raft_consensus.cc:482] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.044571 4462 leader_election.cc:302] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:34.045665 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 113088us
I20240503 02:12:34.045753 5868 raft_consensus.cc:2781] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:34.046041 4746 leader_election.cc:341] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.046412 4746 leader_election.cc:302] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.047708 5106 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:34.051682 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 107693us
I20240503 02:12:34.054023 6013 raft_consensus.cc:504] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.058440 4810 raft_consensus.cc:2454] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:34.059051 6035 raft_consensus.cc:2726] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.059039 4608 leader_election.cc:302] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.059350 5851 raft_consensus.cc:2781] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.059290 6001 raft_consensus.cc:504] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.059480 5851 raft_consensus.cc:686] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:34.059903 5851 consensus_queue.cc:237] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.060129 5106 tablet_bootstrap.cc:654] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.060238 6001 leader_election.cc:288] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.066277 6024 raft_consensus.cc:482] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.066496 6024 raft_consensus.cc:504] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.067430 6024 leader_election.cc:288] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.076335 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:34.076514 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c3710b653aa24b7ca389aa95d7cdd2fe
I20240503 02:12:34.077000 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 211dfd4049684f5e8af927b2a19263b7
I20240503 02:12:34.077934 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 50e04292742646049f39b6aa29338d75
I20240503 02:12:34.079393 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6e635f283288406aa16b3941be64e777
I20240503 02:12:34.079530 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_61c9ba66e720449da2d10423b0c029bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.079990 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 6e635f283288406aa16b3941be64e777 deleted and replaced by tablet cb2c0539bc1c4140b09ac51ad670c0e5
W20240503 02:12:34.080413 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61c9ba66e720449da2d10423b0c029bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.080159 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_278fc530c46a46159b5cff1d43aa3a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.081678 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.082549 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23f94387862b4b6784389cd83a9ac77e" } new_table_name: "default.Table_fe7f91fccadf4fa482c48523fb77a990" modify_external_catalogs: true
I20240503 02:12:34.081810 5828 raft_consensus.cc:504] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.084264 4808 raft_consensus.cc:2454] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.084702 5159 ts_tablet_manager.cc:1888] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:34.085084 4608 leader_election.cc:302] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.085760 6024 raft_consensus.cc:2781] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.085886 6024 raft_consensus.cc:482] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.086002 6024 raft_consensus.cc:3037] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.077883 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:34.086681 4810 raft_consensus.cc:3037] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.089375 5828 leader_election.cc:288] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.089669 5159 log.cc:1196] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/34eb69d5f4bd427a8e430a00fb076f83
W20240503 02:12:34.091100 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 44713.4us
I20240503 02:12:34.093912 5159 ts_tablet_manager.cc:1909] T 34eb69d5f4bd427a8e430a00fb076f83 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:34.096679 5828 raft_consensus.cc:482] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.096830 5828 raft_consensus.cc:504] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.097508 5828 leader_election.cc:288] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:34.097944 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396b48865e954d05b9ba8084a2e7f29b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.098153 4668 raft_consensus.cc:2454] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.099660 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.101014 5106 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.101642 5106 ts_tablet_manager.cc:1367] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.000s
I20240503 02:12:34.102669 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.104171 4746 leader_election.cc:302] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.103715 5106 raft_consensus.cc:348] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.104640 5106 raft_consensus.cc:374] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.104727 5106 raft_consensus.cc:729] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.105017 5106 consensus_queue.cc:260] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.106024 5106 ts_tablet_manager.cc:1398] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:34.106372 5106 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.108130 5106 tablet_bootstrap.cc:654] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.111790 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4469969f6dd045b5834db5a53c4ff0cb" } new_table_name: "default.Table_56845fb712d74923ac2ad61b5a1f6e3e" modify_external_catalogs: true
I20240503 02:12:34.113943 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d6eeea17720743c8b78544cd9e6efefd" } new_table_name: "default.Table_83898e39e7794bc582e22bab8a8e0e12" modify_external_catalogs: true
W20240503 02:12:34.116833 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_278fc530c46a46159b5cff1d43aa3a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.120190 6042 raft_consensus.cc:2781] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.120311 6042 raft_consensus.cc:482] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.120378 6042 raft_consensus.cc:3037] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:34.120218 4746 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:34.120750 4746 leader_election.cc:341] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.123662 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b21a3010a4470288fbdd684ef64e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.124996 5105 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.125178 5105 ts_tablet_manager.cc:1367] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.323s user 0.009s sys 0.006s
I20240503 02:12:34.125828 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.126130 5105 raft_consensus.cc:348] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.126389 5105 raft_consensus.cc:374] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.126463 5105 raft_consensus.cc:729] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.126448 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b21a3010a4470288fbdd684ef64e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.126772 5105 consensus_queue.cc:260] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.127498 5105 ts_tablet_manager.cc:1398] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:34.127720 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dfd4049684f5e8af927b2a19263b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.127889 5105 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.130028 6013 leader_election.cc:288] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.132862 5105 tablet_bootstrap.cc:654] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.135200 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.138772 5105 tablet_bootstrap.cc:492] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:34.138796 4462 leader_election.cc:341] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.138922 5105 ts_tablet_manager.cc:1367] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:12:34.140291 5105 raft_consensus.cc:348] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.140514 5105 raft_consensus.cc:374] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.140578 5105 raft_consensus.cc:729] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.140786 5105 consensus_queue.cc:260] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.141191 5106 tablet_bootstrap.cc:492] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.141330 5106 ts_tablet_manager.cc:1367] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20240503 02:12:34.141578 5105 ts_tablet_manager.cc:1398] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.141973 5105 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.142161 5106 raft_consensus.cc:348] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.142335 5106 raft_consensus.cc:374] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.142382 5106 raft_consensus.cc:729] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.142602 5106 consensus_queue.cc:260] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.143092 5106 ts_tablet_manager.cc:1398] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:34.143342 5106 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.137218 5873 leader_election.cc:288] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.145516 5106 tablet_bootstrap.cc:654] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.148739 5999 raft_consensus.cc:2726] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.148834 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396b48865e954d05b9ba8084a2e7f29b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.152752 5105 tablet_bootstrap.cc:654] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.155347 5106 tablet_bootstrap.cc:492] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.155541 5106 ts_tablet_manager.cc:1367] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 02:12:34.157001 5106 raft_consensus.cc:348] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.157200 5106 raft_consensus.cc:374] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.157255 5106 raft_consensus.cc:729] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.148408 4462 leader_election.cc:302] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:34.157457 5106 consensus_queue.cc:260] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.158217 5106 ts_tablet_manager.cc:1398] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:34.158497 5106 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.160441 5106 tablet_bootstrap.cc:654] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.162751 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.163039 4522 raft_consensus.cc:2454] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
W20240503 02:12:34.163686 4746 leader_election.cc:341] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.164283 4746 leader_election.cc:302] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:34.164628 5828 raft_consensus.cc:2781] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.164748 5828 raft_consensus.cc:482] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.164829 5828 raft_consensus.cc:3037] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.166318 5105 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.166497 5105 ts_tablet_manager.cc:1367] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20240503 02:12:34.167428 5105 raft_consensus.cc:348] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.167693 5105 raft_consensus.cc:374] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.167764 5105 raft_consensus.cc:729] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.167609 5990 raft_consensus.cc:504] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.168226 5105 consensus_queue.cc:260] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.169072 5105 ts_tablet_manager.cc:1398] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:34.169381 5105 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.169338 5873 raft_consensus.cc:2726] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.169695 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.169836 5106 tablet_bootstrap.cc:492] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.169962 5106 ts_tablet_manager.cc:1367] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
W20240503 02:12:34.170758 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 92029.6us
I20240503 02:12:34.170737 5106 raft_consensus.cc:348] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.170909 5106 raft_consensus.cc:374] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.170959 5106 raft_consensus.cc:729] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.171586 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.171648 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ea1eb32394b04b72258d24196202b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.171852 4807 raft_consensus.cc:2454] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:34.171955 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ea1eb32394b04b72258d24196202b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.171650 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.172111 4670 raft_consensus.cc:2454] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:34.172343 5990 leader_election.cc:288] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.171140 5106 consensus_queue.cc:260] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.173117 5106 ts_tablet_manager.cc:1398] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.173395 5106 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.175267 5106 tablet_bootstrap.cc:654] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.175586 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:34.176059 4746 leader_election.cc:341] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.176267 5105 tablet_bootstrap.cc:654] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.176424 4525 raft_consensus.cc:2454] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.177474 4746 leader_election.cc:302] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:34.177970 4522 raft_consensus.cc:1261] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:12:34.178298 5990 raft_consensus.cc:2781] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.178427 5990 raft_consensus.cc:482] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.178493 5990 raft_consensus.cc:3037] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.178493 6035 consensus_queue.cc:1035] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:12:34.179525 4608 leader_election.cc:341] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.179927 4670 raft_consensus.cc:3037] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.180799 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19fb0929bd234788b5313d96d2e4845c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.180871 4462 leader_election.cc:302] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.180999 4522 raft_consensus.cc:3037] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:34.181159 4462 leader_election.cc:341] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.181330 4462 leader_election.cc:302] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:34.181933 4462 leader_election.cc:341] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.182018 4462 leader_election.cc:302] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:34.182281 5873 raft_consensus.cc:2781] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.182443 6013 raft_consensus.cc:2781] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.182541 6013 raft_consensus.cc:482] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.182633 6013 raft_consensus.cc:3037] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.182600 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.182771 4525 raft_consensus.cc:3037] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.183765 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.187330 5106 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.187534 5106 ts_tablet_manager.cc:1367] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 02:12:34.188601 5106 raft_consensus.cc:348] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.188856 5106 raft_consensus.cc:374] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.188906 5106 raft_consensus.cc:729] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.189193 5106 consensus_queue.cc:260] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.189785 5106 ts_tablet_manager.cc:1398] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:34.190066 5106 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.191785 5106 tablet_bootstrap.cc:654] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.195067 6045 raft_consensus.cc:2726] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.195294 6045 consensus_queue.cc:1035] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.196643 5105 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.196902 5105 ts_tablet_manager.cc:1367] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20240503 02:12:34.197953 5105 raft_consensus.cc:348] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.198346 5105 raft_consensus.cc:374] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.198460 5105 raft_consensus.cc:729] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.198940 5105 consensus_queue.cc:260] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.199879 5105 ts_tablet_manager.cc:1398] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:34.200488 5873 raft_consensus.cc:686] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:34.200763 5873 consensus_queue.cc:237] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.201364 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 535) took 1989 ms. Trace:
I20240503 02:12:34.201459 5115 rpcz_store.cc:276] 0503 02:12:32.212132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.212447 (+ 315us) service_pool.cc:225] Handling call
0503 02:12:33.601686 (+1389239us) ts_tablet_manager.cc:1120] Deleting tablet dedead5a2bf347f6991cce8e9b4db708
0503 02:12:33.601698 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.858873 (+257175us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:33.978807 (+119934us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.201326 (+222519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377353}
I20240503 02:12:34.201961 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 536) took 1989 ms. Trace:
I20240503 02:12:34.202046 5115 rpcz_store.cc:276] 0503 02:12:32.212326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.212490 (+ 164us) service_pool.cc:225] Handling call
0503 02:12:34.201786 (+1989296us) ts_tablet_manager.cc:1120] Deleting tablet 3478033c4ed94268a2436a2942c27a68
0503 02:12:34.201798 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.201941 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1972293}
I20240503 02:12:34.202412 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 543) took 1976 ms. Trace:
I20240503 02:12:34.202467 5115 rpcz_store.cc:276] 0503 02:12:32.225960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.227253 (+ 1293us) service_pool.cc:225] Handling call
0503 02:12:34.202319 (+1975066us) ts_tablet_manager.cc:1120] Deleting tablet d271d2c112ba44b68d2933a6ebfe7a18
0503 02:12:34.202326 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.202401 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967753}
I20240503 02:12:34.202716 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 545) took 1976 ms. Trace:
I20240503 02:12:34.202766 5115 rpcz_store.cc:276] 0503 02:12:32.226151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.227324 (+ 1173us) service_pool.cc:225] Handling call
0503 02:12:34.202639 (+1975315us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:34.202646 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.202706 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967438}
I20240503 02:12:34.202999 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 542) took 1977 ms. Trace:
I20240503 02:12:34.204332 5115 rpcz_store.cc:276] 0503 02:12:32.225839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.228476 (+ 2637us) service_pool.cc:225] Handling call
0503 02:12:34.202925 (+1974449us) ts_tablet_manager.cc:1120] Deleting tablet 5b91a3f3cad74f6aa6035a013cca651f
0503 02:12:34.202931 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.202990 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1966301}
I20240503 02:12:34.204577 5115 tablet_replica.cc:330] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:34.204751 5115 raft_consensus.cc:2229] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:34.204895 5115 raft_consensus.cc:2258] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.205508 5115 ts_tablet_manager.cc:1875] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:34.208164 5105 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.211122 5105 tablet_bootstrap.cc:654] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.213488 4608 leader_election.cc:341] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.219342 6056 tablet.cc:1722] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:34.219976 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b21a3010a4470288fbdd684ef64e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.222344 4668 raft_consensus.cc:3037] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:34.224460 5105 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.224714 5105 ts_tablet_manager.cc:1367] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:12:34.226501 5105 raft_consensus.cc:348] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.226879 5105 raft_consensus.cc:374] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.226997 5105 raft_consensus.cc:729] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.227303 5105 consensus_queue.cc:260] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.228833 5106 tablet_bootstrap.cc:492] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.229015 5106 ts_tablet_manager.cc:1367] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.004s
I20240503 02:12:34.231195 5106 raft_consensus.cc:348] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.231389 5106 raft_consensus.cc:374] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.231460 5106 raft_consensus.cc:729] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.231683 5106 consensus_queue.cc:260] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.232431 5106 ts_tablet_manager.cc:1398] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:34.232719 5106 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.233673 5103 tablet_bootstrap.cc:492] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.233816 5103 ts_tablet_manager.cc:1367] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.409s user 0.028s sys 0.076s
I20240503 02:12:34.234163 6045 consensus_queue.cc:1035] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.234323 5106 tablet_bootstrap.cc:654] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.235021 5868 raft_consensus.cc:686] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:34.235370 5868 consensus_queue.cc:237] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.237000 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dfd4049684f5e8af927b2a19263b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.237655 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 211dfd4049684f5e8af927b2a19263b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.238273 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e04292742646049f39b6aa29338d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.238487 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e04292742646049f39b6aa29338d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.243230 5106 tablet_bootstrap.cc:492] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.234540 5103 raft_consensus.cc:348] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.243322 5103 raft_consensus.cc:374] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.243377 5106 ts_tablet_manager.cc:1367] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 02:12:34.243438 5103 raft_consensus.cc:729] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.243688 5103 consensus_queue.cc:260] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.244361 5103 ts_tablet_manager.cc:1398] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 02:12:34.244649 5103 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.246073 5103 tablet_bootstrap.cc:654] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.249498 5105 ts_tablet_manager.cc:1398] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:12:34.249673 4522 raft_consensus.cc:2454] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:34.250216 4608 leader_election.cc:302] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.250509 5999 raft_consensus.cc:2781] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.250660 5999 raft_consensus.cc:686] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:34.249879 5105 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.250973 5999 consensus_queue.cc:237] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.254015 4525 raft_consensus.cc:2454] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:34.254612 4746 leader_election.cc:302] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:34.255039 6057 raft_consensus.cc:2781] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.255188 6057 raft_consensus.cc:686] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:34.255453 6057 consensus_queue.cc:237] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.256902 5114 ts_tablet_manager.cc:1888] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:34.257066 5114 log.cc:1196] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/bbab4368e73b4944b9853c456a4db7bd
I20240503 02:12:34.257524 5114 ts_tablet_manager.cc:1909] T bbab4368e73b4944b9853c456a4db7bd P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:34.259270 4810 raft_consensus.cc:2454] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:34.262097 6024 raft_consensus.cc:504] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.262769 6024 leader_election.cc:288] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.263484 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:34.267529 4608 leader_election.cc:341] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.267848 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:34.268044 4810 raft_consensus.cc:3037] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.268662 5105 tablet_bootstrap.cc:654] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.273598 5105 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.273877 5105 ts_tablet_manager.cc:1367] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240503 02:12:34.275120 5105 raft_consensus.cc:348] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.275436 5105 raft_consensus.cc:374] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.275543 5105 raft_consensus.cc:729] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.275923 5105 consensus_queue.cc:260] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.278311 5105 ts_tablet_manager.cc:1398] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:34.278723 5105 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:34.278724 4260 consensus_peers.cc:485] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d -> Peer 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689): Couldn't send request to peer 145afaa7548f4b7fafb23f20276f0cb4. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:12:34.279830 5991 raft_consensus.cc:482] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.279860 5999 raft_consensus.cc:482] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.279978 5999 raft_consensus.cc:504] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.280032 5991 raft_consensus.cc:504] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.280663 5991 leader_election.cc:288] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.280730 5999 leader_election.cc:288] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.281775 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.282025 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.282238 4525 raft_consensus.cc:2454] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.282541 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.282500 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.282701 4522 raft_consensus.cc:2454] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.285986 5105 tablet_bootstrap.cc:654] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.286007 4608 leader_election.cc:302] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.286329 5999 raft_consensus.cc:2781] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.286525 5999 raft_consensus.cc:482] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.286643 5999 raft_consensus.cc:3037] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.287478 4746 leader_election.cc:302] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.287752 5991 raft_consensus.cc:2781] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.287822 5991 raft_consensus.cc:482] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.287876 5991 raft_consensus.cc:3037] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.290199 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 540) took 4044 ms. Trace:
I20240503 02:12:34.290331 5159 rpcz_store.cc:276] 0503 02:12:30.245275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.245362 (+ 87us) service_pool.cc:225] Handling call
0503 02:12:33.769255 (+3523893us) ts_tablet_manager.cc:1120] Deleting tablet 34eb69d5f4bd427a8e430a00fb076f83
0503 02:12:33.769270 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.952252 (+182982us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.084576 (+132324us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.290109 (+205533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3519590}
I20240503 02:12:34.291028 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 546) took 3313 ms. Trace:
I20240503 02:12:34.291110 5159 rpcz_store.cc:276] 0503 02:12:30.977762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:30.977820 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:34.290843 (+3313023us) ts_tablet_manager.cc:1120] Deleting tablet 5b91a3f3cad74f6aa6035a013cca651f
0503 02:12:34.290859 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.291007 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2746332}
I20240503 02:12:34.291473 5159 tablet_replica.cc:330] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:34.315588 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e04292742646049f39b6aa29338d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.333503 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e635f283288406aa16b3941be64e777 with delete_type TABLET_DATA_DELETED (replaced by tablet cb2c0539bc1c4140b09ac51ad670c0e5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.333899 6024 raft_consensus.cc:482] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.334062 6024 raft_consensus.cc:504] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.335177 6024 leader_election.cc:288] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.336179 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.336480 4807 raft_consensus.cc:2454] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.337354 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.337476 4608 leader_election.cc:302] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:34.337980 4608 leader_election.cc:341] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.338485 6024 raft_consensus.cc:2781] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.338649 6024 raft_consensus.cc:482] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.338793 6024 raft_consensus.cc:3037] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.340047 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e635f283288406aa16b3941be64e777 with delete_type TABLET_DATA_DELETED (replaced by tablet cb2c0539bc1c4140b09ac51ad670c0e5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.340987 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e635f283288406aa16b3941be64e777 with delete_type TABLET_DATA_DELETED (replaced by tablet cb2c0539bc1c4140b09ac51ad670c0e5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.341763 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.342181 5105 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.342406 5105 ts_tablet_manager.cc:1367] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.005s
I20240503 02:12:34.342811 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.343542 5105 raft_consensus.cc:348] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.343842 5105 raft_consensus.cc:374] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.345007 6013 raft_consensus.cc:504] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.345008 5999 raft_consensus.cc:504] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.345772 5999 leader_election.cc:288] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.352815 5999 raft_consensus.cc:482] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.353101 5999 raft_consensus.cc:504] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.354034 5999 leader_election.cc:288] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.358109 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ea1eb32394b04b72258d24196202b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:34.358420 4673 raft_consensus.cc:3037] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.361337 5105 raft_consensus.cc:729] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.365614 5105 consensus_queue.cc:260] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.366468 5999 raft_consensus.cc:482] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.366637 5999 raft_consensus.cc:504] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.367424 5999 leader_election.cc:288] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.359678 6042 raft_consensus.cc:504] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.370820 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 466) took 6689 ms. Trace:
I20240503 02:12:34.370985 5114 rpcz_store.cc:276] 0503 02:12:27.683000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.683098 (+ 98us) service_pool.cc:225] Handling call
0503 02:12:33.758000 (+6074902us) ts_tablet_manager.cc:1120] Deleting tablet bbab4368e73b4944b9853c456a4db7bd
0503 02:12:33.758009 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:33.852906 (+ 94897us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.256784 (+403878us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.370779 (+113995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5614426}
I20240503 02:12:34.371646 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 474) took 6182 ms. Trace:
I20240503 02:12:34.375749 5105 ts_tablet_manager.cc:1398] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.026s user 0.002s sys 0.002s
I20240503 02:12:34.376322 5105 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.397119 6035 consensus_queue.cc:1035] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240503 02:12:34.403198 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396b48865e954d05b9ba8084a2e7f29b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.403226 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396b48865e954d05b9ba8084a2e7f29b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:34.403496 4672 raft_consensus.cc:3037] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.410573 6042 leader_election.cc:288] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.414569 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.414820 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.415081 4527 raft_consensus.cc:2379] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:34.414999 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.414827 4529 raft_consensus.cc:3037] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.415568 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbbfa2d1a49400c809d5b476036cdcd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.416118 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.418046 5105 tablet_bootstrap.cc:654] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.418191 4608 leader_election.cc:341] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.418463 4608 leader_election.cc:302] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.418895 5999 raft_consensus.cc:3037] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.413249 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:34.423965 4811 raft_consensus.cc:2454] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.429198 4608 leader_election.cc:302] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.430979 5851 raft_consensus.cc:2781] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.431087 5851 raft_consensus.cc:482] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.431140 5851 raft_consensus.cc:3037] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.436277 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]) successfully deleted
W20240503 02:12:34.436729 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3478033c4ed94268a2436a2942c27a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3478033c4ed94268a2436a2942c27a68 already in progress: creating tablet
I20240503 02:12:34.441861 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b2768a724a5944449a276cbef0d23e9f" } new_table_name: "default.Table_54f7e670055747f7862a072acefa597e" modify_external_catalogs: true
I20240503 02:12:34.442203 6013 leader_election.cc:288] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.442380 4525 raft_consensus.cc:3037] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.443392 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f5fd27023b3f47f281dfb5474ea3a4c6
I20240503 02:12:34.444352 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5ad66186bf0948d996f2098e51b3a661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.445138 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad66186bf0948d996f2098e51b3a661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.450568 4672 raft_consensus.cc:2454] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:34.457434 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ea1eb32394b04b72258d24196202b" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:34.371744 5114 rpcz_store.cc:276] 0503 02:12:28.189302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.189390 (+ 88us) service_pool.cc:225] Handling call
0503 02:12:34.371444 (+6182054us) ts_tablet_manager.cc:1120] Deleting tablet 0af50eb2b1c949a0a6a71e64a894e8a5
0503 02:12:34.371460 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.371623 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6181780}
I20240503 02:12:34.464533 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet b8474c841cb64373a223ad76b43ecf5f from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:34.464761 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 469) took 6736 ms. Trace:
I20240503 02:12:34.464843 5114 rpcz_store.cc:276] 0503 02:12:27.728268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:27.728340 (+ 72us) service_pool.cc:225] Handling call
0503 02:12:34.464628 (+6736288us) ts_tablet_manager.cc:1120] Deleting tablet aaa77e6c9a494fa2bdc74c5d725624cb
0503 02:12:34.464638 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.464742 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6274679}
I20240503 02:12:34.465763 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 475) took 6276 ms. Trace:
I20240503 02:12:34.465837 5114 rpcz_store.cc:276] 0503 02:12:28.189506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.189561 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:34.465651 (+6276090us) ts_tablet_manager.cc:1120] Deleting tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3
0503 02:12:34.465660 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.465747 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6270289}
I20240503 02:12:34.466120 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 476) took 6276 ms. Trace:
I20240503 02:12:34.466194 5114 rpcz_store.cc:276] 0503 02:12:28.189619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.189774 (+ 155us) service_pool.cc:225] Handling call
0503 02:12:34.466018 (+6276244us) ts_tablet_manager.cc:1120] Deleting tablet 367de4ff0a6f4abfa2924a070f387cbd
0503 02:12:34.466026 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.466107 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6269943}
I20240503 02:12:34.466461 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 478) took 6259 ms. Trace:
I20240503 02:12:34.466540 5114 rpcz_store.cc:276] 0503 02:12:28.207204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.207254 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:34.466380 (+6259126us) ts_tablet_manager.cc:1120] Deleting tablet 1ce9d746acfe4e9aaea600293470d373
0503 02:12:34.466387 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.466447 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6228822}
I20240503 02:12:34.466871 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 477) took 6265 ms. Trace:
I20240503 02:12:34.467644 4811 raft_consensus.cc:3037] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.468443 5105 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.466943 5114 rpcz_store.cc:276] 0503 02:12:28.201555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.217374 (+ 15819us) service_pool.cc:225] Handling call
0503 02:12:34.466785 (+6249411us) ts_tablet_manager.cc:1120] Deleting tablet 620204f7cb35442f87eae45f84beeee1
0503 02:12:34.466794 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.466858 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5636810}
I20240503 02:12:34.472126 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 486) took 6137 ms. Trace:
I20240503 02:12:34.472218 5114 rpcz_store.cc:276] 0503 02:12:28.334661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:28.418806 (+ 84145us) service_pool.cc:225] Handling call
0503 02:12:34.471987 (+6053181us) ts_tablet_manager.cc:1120] Deleting tablet 5a915d760ba14a31895a5dc7ec5b34fd
0503 02:12:34.472000 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.472107 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5622549}
I20240503 02:12:34.472625 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 493) took 6137 ms. Trace:
I20240503 02:12:34.479383 5114 rpcz_store.cc:276] 0503 02:12:28.335529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:29.083633 (+748104us) service_pool.cc:225] Handling call
0503 02:12:34.472505 (+5388872us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:34.472516 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.472607 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4517649}
I20240503 02:12:34.480188 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 520) took 5628 ms. Trace:
I20240503 02:12:34.480306 5114 rpcz_store.cc:276] 0503 02:12:28.851360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.523248 (+2671888us) service_pool.cc:225] Handling call
0503 02:12:34.480022 (+2956774us) ts_tablet_manager.cc:1120] Deleting tablet dbfe6b2b9cab4517ae7cabec4796c9ce
0503 02:12:34.480037 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.480165 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2956552}
I20240503 02:12:34.481220 5114 tablet_replica.cc:330] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:34.481504 5114 raft_consensus.cc:2229] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:34.483700 5105 ts_tablet_manager.cc:1367] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.010s
I20240503 02:12:34.485687 5114 raft_consensus.cc:2258] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.489069 4810 raft_consensus.cc:2454] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:34.489499 5105 raft_consensus.cc:348] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.491035 4746 leader_election.cc:302] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.491077 5105 raft_consensus.cc:374] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.491225 5105 raft_consensus.cc:729] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.492235 4608 leader_election.cc:302] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.492774 6001 raft_consensus.cc:2781] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.493144 6035 raft_consensus.cc:2781] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.493425 6001 raft_consensus.cc:686] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:34.493688 6001 consensus_queue.cc:237] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.496026 5105 consensus_queue.cc:260] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.499035 6035 raft_consensus.cc:686] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:34.499445 6035 consensus_queue.cc:237] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.499996 5114 ts_tablet_manager.cc:1875] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:34.437686 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d271d2c112ba44b68d2933a6ebfe7a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d271d2c112ba44b68d2933a6ebfe7a18 already in progress: creating tablet
W20240503 02:12:34.505864 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 68196.5us
W20240503 02:12:34.510385 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 48e713ee37d042309166b0f118e14986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e713ee37d042309166b0f118e14986 already in progress: creating tablet
W20240503 02:12:34.510900 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5b91a3f3cad74f6aa6035a013cca651f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b91a3f3cad74f6aa6035a013cca651f already in progress: creating tablet
I20240503 02:12:34.511289 5103 tablet_bootstrap.cc:492] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.511502 5103 ts_tablet_manager.cc:1367] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.267s user 0.003s sys 0.000s
I20240503 02:12:34.511798 6035 raft_consensus.cc:482] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.511988 6035 raft_consensus.cc:504] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.513126 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19ed19da5b74583bfc3b21d88e937c9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.513379 4672 raft_consensus.cc:2454] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.514331 4746 leader_election.cc:302] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.514595 6078 raft_consensus.cc:2781] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.515172 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19ed19da5b74583bfc3b21d88e937c9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.515414 5106 raft_consensus.cc:348] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.515610 5106 raft_consensus.cc:374] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.515669 5106 raft_consensus.cc:729] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:34.515774 4746 leader_election.cc:341] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.515937 6035 leader_election.cc:288] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.515864 5106 consensus_queue.cc:260] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.516403 4675 raft_consensus.cc:2454] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.516414 5106 ts_tablet_manager.cc:1398] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.273s user 0.001s sys 0.001s
I20240503 02:12:34.516975 5106 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.514714 6078 raft_consensus.cc:482] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.517437 6078 raft_consensus.cc:3037] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.518888 5106 tablet_bootstrap.cc:654] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.520648 5103 raft_consensus.cc:348] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.520838 5103 raft_consensus.cc:374] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.524601 5105 ts_tablet_manager.cc:1398] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.041s user 0.004s sys 0.008s
I20240503 02:12:34.525681 5105 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.526100 5106 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.526254 5106 ts_tablet_manager.cc:1367] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:34.527127 5106 raft_consensus.cc:348] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.527319 5106 raft_consensus.cc:374] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.527374 5106 raft_consensus.cc:729] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.527586 5106 consensus_queue.cc:260] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.527864 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 34eb69d5f4bd427a8e430a00fb076f83 (table default.Table_a6d935ff1f554a62a312091e6252d15a [id=7118c326733f4c6695909797bd035f9e]) successfully deleted
W20240503 02:12:34.528115 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5b91a3f3cad74f6aa6035a013cca651f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b91a3f3cad74f6aa6035a013cca651f already in progress: creating tablet
I20240503 02:12:34.528358 5106 ts_tablet_manager.cc:1398] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:34.528625 5106 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.528957 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet bbab4368e73b4944b9853c456a4db7bd (table default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]) successfully deleted
W20240503 02:12:34.529259 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af50eb2b1c949a0a6a71e64a894e8a5 already in progress: creating tablet
W20240503 02:12:34.529531 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet aaa77e6c9a494fa2bdc74c5d725624cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaa77e6c9a494fa2bdc74c5d725624cb already in progress: creating tablet
W20240503 02:12:34.529753 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 already in progress: creating tablet
W20240503 02:12:34.529929 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 367de4ff0a6f4abfa2924a070f387cbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 367de4ff0a6f4abfa2924a070f387cbd already in progress: creating tablet
W20240503 02:12:34.530072 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1ce9d746acfe4e9aaea600293470d373 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce9d746acfe4e9aaea600293470d373 already in progress: creating tablet
W20240503 02:12:34.530254 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 620204f7cb35442f87eae45f84beeee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 620204f7cb35442f87eae45f84beeee1 already in progress: creating tablet
I20240503 02:12:34.530279 5106 tablet_bootstrap.cc:654] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.530535 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5a915d760ba14a31895a5dc7ec5b34fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a915d760ba14a31895a5dc7ec5b34fd already in progress: creating tablet
I20240503 02:12:34.520891 5103 raft_consensus.cc:729] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.531495 5103 consensus_queue.cc:260] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.533231 5103 ts_tablet_manager.cc:1398] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:12:34.534560 5103 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.539974 5103 tablet_bootstrap.cc:654] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.542624 5106 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:34.543133 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 607e49cde2fd4aab9f44921d3c3921a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607e49cde2fd4aab9f44921d3c3921a5 already in progress: creating tablet
I20240503 02:12:34.543968 5106 ts_tablet_manager.cc:1367] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:12:34.545115 5106 raft_consensus.cc:348] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:34.545650 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet dbfe6b2b9cab4517ae7cabec4796c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfe6b2b9cab4517ae7cabec4796c9ce already in progress: creating tablet
I20240503 02:12:34.548370 4525 raft_consensus.cc:2454] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
W20240503 02:12:34.562513 5937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:34.563062 5937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:34.563305 5937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:34.570710 5103 tablet_bootstrap.cc:492] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.573658 5828 raft_consensus.cc:504] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.574502 5828 leader_election.cc:288] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.574657 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.574831 5159 raft_consensus.cc:2229] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:34.574987 5105 tablet_bootstrap.cc:654] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.571290 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.575927 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.576119 4525 raft_consensus.cc:3037] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.576592 5103 ts_tablet_manager.cc:1367] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240503 02:12:34.547442 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fd27023b3f47f281dfb5474ea3a4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.548492 4668 raft_consensus.cc:1261] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 02:12:34.549134 4642 consensus_meta.cc:287] T 7c19263f06ec41d7a6fe6a8df7f88f8f P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.658s user 0.002s sys 0.003s
W20240503 02:12:34.549271 4647 consensus_meta.cc:287] T b30e440f79a64e04b7e527b52299301f P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.653s user 0.007s sys 0.000s
W20240503 02:12:34.577744 5937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20240503 02:12:34.577698 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:34.577863 5937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:34.577936 5937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:34.578015 5937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:34.567916 5115 ts_tablet_manager.cc:1888] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:34.578289 5115 log.cc:1196] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/34eb69d5f4bd427a8e430a00fb076f83
I20240503 02:12:34.578747 5115 ts_tablet_manager.cc:1909] T 34eb69d5f4bd427a8e430a00fb076f83 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:34.578936 5106 raft_consensus.cc:374] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:34.581169 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fa25fee059b46ec915bcc1a01cb8b27 consensus state has no leader
I20240503 02:12:34.583899 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d50112d374614b09ab8cdc93c129eac2" } new_table_name: "default.Table_83329138bcd24006aa233387b43098a7" modify_external_catalogs: true
W20240503 02:12:34.584223 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84f6279f16dc42d69e794e3c6586163d consensus state has no leader
I20240503 02:12:34.584806 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.585610 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.585866 5159 raft_consensus.cc:2258] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.586591 5159 ts_tablet_manager.cc:1875] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:34.587531 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_04087507235042c2831d1475b54a2eb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.588147 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04087507235042c2831d1475b54a2eb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.589676 6077 consensus_queue.cc:1035] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:12:34.589883 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee16f1beb6a34a85bc872dedd64472ce consensus state has no leader
W20240503 02:12:34.589844 4746 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.590304 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_66c2a4cc5ab648c5938e6aecd6792779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.590760 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66c2a4cc5ab648c5938e6aecd6792779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.580844 6075 consensus_queue.cc:1035] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.596278 5937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:43321,127.4.17.125:46221,127.4.17.124:46689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.17.124
--webserver_port=46343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:34.599424 5937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:34.600065 5106 raft_consensus.cc:729] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.601161 5106 consensus_queue.cc:260] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.602208 5103 raft_consensus.cc:348] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.602418 5103 raft_consensus.cc:374] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.602497 5103 raft_consensus.cc:729] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.603003 5106 ts_tablet_manager.cc:1398] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20240503 02:12:34.603327 5106 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.605597 5937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:34.609953 5103 consensus_queue.cc:260] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.605901 6001 consensus_queue.cc:1035] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.614403 5105 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.614574 5105 ts_tablet_manager.cc:1367] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.008s
I20240503 02:12:34.616394 5105 raft_consensus.cc:348] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.616621 5105 raft_consensus.cc:374] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.616725 5105 raft_consensus.cc:729] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.620589 5105 consensus_queue.cc:260] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.621691 5105 ts_tablet_manager.cc:1398] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:12:34.624485 5105 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.625006 5103 ts_tablet_manager.cc:1398] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:12:34.625591 6075 raft_consensus.cc:482] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.625811 6075 raft_consensus.cc:504] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.626597 5105 tablet_bootstrap.cc:654] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.626681 6075 leader_election.cc:288] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.627825 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.628067 4807 raft_consensus.cc:2454] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.628837 4608 leader_election.cc:302] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.629560 6075 raft_consensus.cc:2781] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.629690 6075 raft_consensus.cc:482] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.629812 6075 raft_consensus.cc:3037] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:34.630511 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 49611us
I20240503 02:12:34.629069 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:34.631567 4608 leader_election.cc:341] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.634001 5103 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.641778 5105 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.641986 5105 ts_tablet_manager.cc:1367] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20240503 02:12:34.642925 5105 raft_consensus.cc:348] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.643183 5105 raft_consensus.cc:374] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.643283 5105 raft_consensus.cc:729] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.643566 5105 consensus_queue.cc:260] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.644152 5105 ts_tablet_manager.cc:1398] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:34.644534 6077 consensus_queue.cc:1035] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:12:34.645785 5105 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.648382 5105 tablet_bootstrap.cc:654] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.650730 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83a1d5b2de8d4d3ba5190750545c9f7b consensus state has no leader
I20240503 02:12:34.651082 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 84f6279f16dc42d69e794e3c6586163d from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:34.651983 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ce9d746acfe4e9aaea600293470d373 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:34.657600 4784 consensus_meta.cc:287] T 77fa5be5a4f64c189ef619bc38e21eae P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.570s user 0.002s sys 0.002s
I20240503 02:12:34.659657 4273 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
W20240503 02:12:34.662326 6095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:34.664402 4673 raft_consensus.cc:2454] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
W20240503 02:12:34.665027 4636 consensus_meta.cc:287] T ce7b279d0df9438f9e90b01c4e619c7d P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.621s user 0.005s sys 0.001s
W20240503 02:12:34.664855 4639 consensus_meta.cc:287] T e489b424b2d546e2aa993163f13dcbd9 P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.610s user 0.004s sys 0.000s
I20240503 02:12:34.667944 5991 raft_consensus.cc:504] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.668851 5991 leader_election.cc:288] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.668617 4272 catalog_manager.cc:5484] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:34.669095 4272 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:34.670584 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:34.670840 4673 raft_consensus.cc:3037] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.673811 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 72cafc744af34974a84ef4f72b23802c (DEFAULT_TABLE table=default.Table_40eee5fde9014729a9372235d138bda5 [id=978285601e1d49278aecd2b18250a7b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:34.674031 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:34.674400 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cafc744af34974a84ef4f72b23802c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.676017 4786 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f (DEFAULT_TABLE table=default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7 [id=23d27eb8a7674e2387125ea8d625f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.676399 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee32345fb9349f19af09a3447c4caad (DEFAULT_TABLE table=default.Table_e6175ee8a08c4522912024d62f15a984 [id=e60f2596fe074c40bd44f0e358687016]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.676854 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee32345fb9349f19af09a3447c4caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.677309 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 72cafc744af34974a84ef4f72b23802c (DEFAULT_TABLE table=default.Table_40eee5fde9014729a9372235d138bda5 [id=978285601e1d49278aecd2b18250a7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.677939 4647 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f (DEFAULT_TABLE table=default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7 [id=23d27eb8a7674e2387125ea8d625f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.678483 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecddfcfed2436e98bb7b6b38beb03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.681602 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 91e785eaaf534b8c81aa6132ab929eea (DEFAULT_TABLE table=default.Table_a1279c4656b24d92a8a44b6f28a0616e [id=6a065613c1b64ac394822d50135b1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.681825 5991 raft_consensus.cc:482] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.682041 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e785eaaf534b8c81aa6132ab929eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.681982 5991 raft_consensus.cc:504] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.682685 5991 leader_election.cc:288] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.683670 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:34.684988 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee32345fb9349f19af09a3447c4caad (DEFAULT_TABLE table=default.Table_e6175ee8a08c4522912024d62f15a984 [id=e60f2596fe074c40bd44f0e358687016]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:34.685475 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 84f6279f16dc42d69e794e3c6586163d (table default.Table_fe7f91fccadf4fa482c48523fb77a990 [id=70fb3c2002314174a5705603052275df], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 84f6279f16dc42d69e794e3c6586163d consensus state has no leader
I20240503 02:12:34.687234 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1c97fd1492d442e955a652306d0e6a6" } modify_external_catalogs: true
I20240503 02:12:34.687475 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f1c97fd1492d442e955a652306d0e6a6" } modify_external_catalogs: true
I20240503 02:12:34.688076 5937 server_base.cc:1034] running on GCE node
I20240503 02:12:34.689324 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 91e785eaaf534b8c81aa6132ab929eea (DEFAULT_TABLE table=default.Table_a1279c4656b24d92a8a44b6f28a0616e [id=6a065613c1b64ac394822d50135b1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.690064 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:34.690773 4746 leader_election.cc:341] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.691888 5105 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.692028 5105 ts_tablet_manager.cc:1367] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.009s
I20240503 02:12:34.692344 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f471b5f3094048268179ff4ff91b119f" } modify_external_catalogs: true
I20240503 02:12:34.692516 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f471b5f3094048268179ff4ff91b119f" } modify_external_catalogs: true
I20240503 02:12:34.692835 5105 raft_consensus.cc:348] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.693073 5105 raft_consensus.cc:374] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.693167 5105 raft_consensus.cc:729] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.693403 5105 consensus_queue.cc:260] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.693884 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e1aa615a80d47258acd2c493d6cb299" } modify_external_catalogs: true
I20240503 02:12:34.694052 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e1aa615a80d47258acd2c493d6cb299" } modify_external_catalogs: true
I20240503 02:12:34.694857 5105 ts_tablet_manager.cc:1398] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.695780 5105 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.697530 5105 tablet_bootstrap.cc:654] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.699699 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecddfcfed2436e98bb7b6b38beb03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.699967 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e785eaaf534b8c81aa6132ab929eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.700943 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cafc744af34974a84ef4f72b23802c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.701745 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee32345fb9349f19af09a3447c4caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.711280 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b" } modify_external_catalogs: true
I20240503 02:12:34.711501 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_87f2a7e29cc64357a81e05e9d2cd0d0b" } modify_external_catalogs: true
I20240503 02:12:34.711709 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1c52b1bbf50148c5b4753da4f8fe72fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.712273 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c52b1bbf50148c5b4753da4f8fe72fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.713728 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_91951087060348328675a5350c9a1dea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.714215 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91951087060348328675a5350c9a1dea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.711797 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b41dce88919456aa846dbdcf5e2759f from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:34.715453 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b1103caea8f04d32ba60345676c5fa63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.716091 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1103caea8f04d32ba60345676c5fa63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:34.717255 6098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:34.720170 5937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:34.725776 5105 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.725944 5105 ts_tablet_manager.cc:1367] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.015s
I20240503 02:12:34.726799 5105 raft_consensus.cc:348] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.726991 5105 raft_consensus.cc:374] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.727058 5105 raft_consensus.cc:729] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.727278 5105 consensus_queue.cc:260] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.728725 5105 ts_tablet_manager.cc:1398] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:34.729094 5105 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.730710 5105 tablet_bootstrap.cc:654] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.734010 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b8474c841cb64373a223ad76b43ecf5f
I20240503 02:12:34.734021 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a8ce4b33f90c40c48cf0a538a1a7fde4
I20240503 02:12:34.734532 4270 catalog_manager.cc:6554] ReplaceTablet: tablet b8474c841cb64373a223ad76b43ecf5f deleted and replaced by tablet 73198ec769a9486da3f85e8fcbef8e78
I20240503 02:12:34.734776 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 84f6279f16dc42d69e794e3c6586163d
I20240503 02:12:34.734853 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_930427a07e994d4f8ad39f8cd1881e2a" } modify_external_catalogs: true
I20240503 02:12:34.735077 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_930427a07e994d4f8ad39f8cd1881e2a" } modify_external_catalogs: true
I20240503 02:12:34.734953 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_de08435101204e569789eb4f8e2ed689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.735260 4268 catalog_manager.cc:6554] ReplaceTablet: tablet 84f6279f16dc42d69e794e3c6586163d deleted and replaced by tablet f50322e5e3eb4eb3a1b3479fd6c992ea
W20240503 02:12:34.735615 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de08435101204e569789eb4f8e2ed689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.739198 5105 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.739435 5105 ts_tablet_manager.cc:1367] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 02:12:34.739637 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 04d555c8d8844ee49c960a5279485480
I20240503 02:12:34.735265 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8135907b78284d7ca74396cc77dd0612
I20240503 02:12:34.740715 5105 raft_consensus.cc:348] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.741060 5105 raft_consensus.cc:374] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.741187 5105 raft_consensus.cc:729] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.741501 5105 consensus_queue.cc:260] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.742604 5105 ts_tablet_manager.cc:1398] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:34.742920 5105 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.744170 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f6279f16dc42d69e794e3c6586163d with delete_type TABLET_DATA_DELETED (replaced by tablet f50322e5e3eb4eb3a1b3479fd6c992ea) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.744364 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d555c8d8844ee49c960a5279485480 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.745785 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet b8474c841cb64373a223ad76b43ecf5f with delete_type TABLET_DATA_DELETED (replaced by tablet 73198ec769a9486da3f85e8fcbef8e78) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.746150 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.746528 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 8135907b78284d7ca74396cc77dd0612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.748633 5999 raft_consensus.cc:2726] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.748857 5999 raft_consensus.cc:482] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.749018 5999 raft_consensus.cc:504] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.749641 5999 leader_election.cc:288] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:34.749964 4670 consensus_meta.cc:287] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.570s user 0.000s sys 0.005s
I20240503 02:12:34.751943 5991 consensus_queue.cc:1035] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.752048 6001 raft_consensus.cc:482] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.752221 6001 raft_consensus.cc:504] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.752804 4811 raft_consensus.cc:2454] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:34.753245 6001 leader_election.cc:288] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.753672 5159 ts_tablet_manager.cc:1888] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:34.753839 5159 log.cc:1196] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:34.753939 5999 raft_consensus.cc:482] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.754079 5999 raft_consensus.cc:504] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.754217 5159 ts_tablet_manager.cc:1909] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:34.754730 5999 leader_election.cc:288] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:34.757207 4782 consensus_meta.cc:287] T 3c421694ce1e424da7953d061a6bca3e P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.668s user 0.004s sys 0.000s
I20240503 02:12:34.761799 5105 tablet_bootstrap.cc:654] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.763806 4783 consensus_meta.cc:287] T 1c084ab3fedb46d4992f5af2ceede06b P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.652s user 0.003s sys 0.002s
I20240503 02:12:34.657161 4529 raft_consensus.cc:2454] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:34.764613 4780 consensus_meta.cc:287] T b30e440f79a64e04b7e527b52299301f P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.677s user 0.003s sys 0.001s
I20240503 02:12:34.669786 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56701ed6424c60bf1a985eea1936fe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.765079 4519 raft_consensus.cc:3037] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.765118 5103 tablet_bootstrap.cc:654] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.766268 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7447242714c844e685faca5aed71a7f7
W20240503 02:12:34.768364 4793 consensus_meta.cc:287] T d224bded18ed4e64ae52d01c91cacc8f P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.680s user 0.004s sys 0.000s
I20240503 02:12:34.671919 4462 leader_election.cc:302] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:34.769500 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 97610.4us
I20240503 02:12:34.769995 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447242714c844e685faca5aed71a7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.687722 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.770696 6013 raft_consensus.cc:2781] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.770841 6013 raft_consensus.cc:686] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:34.771126 6013 consensus_queue.cc:237] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.691359 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 91e785eaaf534b8c81aa6132ab929eea (DEFAULT_TABLE table=default.Table_a1279c4656b24d92a8a44b6f28a0616e [id=6a065613c1b64ac394822d50135b1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.775261 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e785eaaf534b8c81aa6132ab929eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.691751 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 72cafc744af34974a84ef4f72b23802c (DEFAULT_TABLE table=default.Table_40eee5fde9014729a9372235d138bda5 [id=978285601e1d49278aecd2b18250a7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.780288 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cafc744af34974a84ef4f72b23802c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.692063 4498 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f (DEFAULT_TABLE table=default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7 [id=23d27eb8a7674e2387125ea8d625f5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.784651 5105 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.784637 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecddfcfed2436e98bb7b6b38beb03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.784893 5105 ts_tablet_manager.cc:1367] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.007s
I20240503 02:12:34.785966 5105 raft_consensus.cc:348] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.786212 5105 raft_consensus.cc:374] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.786281 5105 raft_consensus.cc:729] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.786545 5105 consensus_queue.cc:260] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.787232 5105 ts_tablet_manager.cc:1398] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:34.787791 5105 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.692389 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee32345fb9349f19af09a3447c4caad (DEFAULT_TABLE table=default.Table_e6175ee8a08c4522912024d62f15a984 [id=e60f2596fe074c40bd44f0e358687016]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.788547 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee32345fb9349f19af09a3447c4caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.789680 5105 tablet_bootstrap.cc:654] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.791399 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c5d56a6132f94935a81edadcf1f0d7f3" } new_table_name: "default.Table_08e73f7a6e1f427b8311fef7e8821883" modify_external_catalogs: true
I20240503 02:12:34.793947 5991 consensus_queue.cc:1035] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.794715 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_73a988b03f62487b86ace12f89bcd124" } new_table_name: "default.Table_d0e9b6b6436b4eb593d72514890ee977" modify_external_catalogs: true
I20240503 02:12:34.795123 6107 raft_consensus.cc:482] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.795295 6107 raft_consensus.cc:504] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:34.795890 4746 leader_election.cc:341] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.795996 6107 leader_election.cc:288] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.796048 4746 leader_election.cc:302] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.796558 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5904b0f9c29f483eab4730665a8bb2fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.797152 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5904b0f9c29f483eab4730665a8bb2fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.797487 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447242714c844e685faca5aed71a7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.797793 5991 raft_consensus.cc:2726] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.798015 5991 raft_consensus.cc:482] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.798269 5106 tablet_bootstrap.cc:654] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.798161 5991 raft_consensus.cc:504] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.798529 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0d42c9ff3ee949f09e9bd50c70d99727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.798972 5991 leader_election.cc:288] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:34.799031 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d42c9ff3ee949f09e9bd50c70d99727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.799657 5105 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.799803 5105 ts_tablet_manager.cc:1367] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 02:12:34.800572 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_402b0b1faf1e44138b755f83223a5033" } new_table_name: "default.Table_cf5d0a99df5945038f6a923ebeae4b32" modify_external_catalogs: true
I20240503 02:12:34.800720 5105 raft_consensus.cc:348] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.800957 5105 raft_consensus.cc:374] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.801139 6079 raft_consensus.cc:482] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.801431 6079 raft_consensus.cc:504] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.802211 6079 leader_election.cc:288] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.802487 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.802475 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:34.803194 4746 leader_election.cc:341] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:34.803450 4746 leader_election.cc:341] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.803552 4746 leader_election.cc:302] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.804175 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.804401 4670 raft_consensus.cc:2454] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.804837 6079 raft_consensus.cc:2726] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.805385 4746 leader_election.cc:302] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.805760 6079 raft_consensus.cc:2781] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.805848 6079 raft_consensus.cc:482] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.805936 6079 raft_consensus.cc:3037] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.801084 5105 raft_consensus.cc:729] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.806586 5105 consensus_queue.cc:260] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.807289 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:34.807852 4746 leader_election.cc:341] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.808306 5105 ts_tablet_manager.cc:1398] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240503 02:12:34.808660 5105 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.722712 5114 ts_tablet_manager.cc:1888] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:34.745209 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f6279f16dc42d69e794e3c6586163d with delete_type TABLET_DATA_DELETED (replaced by tablet f50322e5e3eb4eb3a1b3479fd6c992ea) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.810703 5114 log.cc:1196] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/4064ff59de474e0db3a578eae10c3fa6
I20240503 02:12:34.746070 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet b8474c841cb64373a223ad76b43ecf5f with delete_type TABLET_DATA_DELETED (replaced by tablet 73198ec769a9486da3f85e8fcbef8e78) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.746404 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.746577 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.746717 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d555c8d8844ee49c960a5279485480 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.810916 5105 tablet_bootstrap.cc:654] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.746848 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 8135907b78284d7ca74396cc77dd0612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.753247 4525 raft_consensus.cc:2454] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:34.756260 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.756407 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.811733 4746 leader_election.cc:302] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
W20240503 02:12:34.811937 4746 raft_consensus.cc:2670] T d271d2c112ba44b68d2933a6ebfe7a18 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:34.812544 5114 ts_tablet_manager.cc:1909] T 4064ff59de474e0db3a578eae10c3fa6 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:34.815665 5937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:34.818871 5937 hybrid_clock.cc:648] HybridClock initialized: now 1714702354818418 us; error 455 us; skew 500 ppm
I20240503 02:12:34.819818 5937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:34.820580 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet cef84b4e6ba24150905994d45ffd0b52
I20240503 02:12:34.822564 5105 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:34.822750 5105 ts_tablet_manager.cc:1367] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20240503 02:12:34.822994 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet cef84b4e6ba24150905994d45ffd0b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:34.823524 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet cef84b4e6ba24150905994d45ffd0b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.823587 5105 raft_consensus.cc:348] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:34.826000 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6c63f889cb54619927ade6989fed63a consensus state has no leader
I20240503 02:12:34.826598 5105 raft_consensus.cc:374] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.826879 5105 raft_consensus.cc:729] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.827276 5105 consensus_queue.cc:260] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.828147 5105 ts_tablet_manager.cc:1398] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:34.828492 5105 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.831275 5103 tablet_bootstrap.cc:492] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.831429 5103 ts_tablet_manager.cc:1367] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.198s user 0.002s sys 0.000s
I20240503 02:12:34.831738 5105 tablet_bootstrap.cc:654] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.832157 5103 raft_consensus.cc:348] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.832384 5103 raft_consensus.cc:374] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.832448 5103 raft_consensus.cc:729] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.833020 5103 consensus_queue.cc:260] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.833158 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.833355 4525 raft_consensus.cc:2454] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.833561 5106 tablet_bootstrap.cc:492] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.833745 5106 ts_tablet_manager.cc:1367] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.230s user 0.003s sys 0.001s
I20240503 02:12:34.835318 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.835598 4810 raft_consensus.cc:2454] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.835642 5103 ts_tablet_manager.cc:1398] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:34.834537 5106 raft_consensus.cc:348] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.836200 5106 raft_consensus.cc:374] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.836333 5106 raft_consensus.cc:729] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.836699 5106 consensus_queue.cc:260] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.837532 5106 ts_tablet_manager.cc:1398] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:34.837925 5106 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.839134 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.839974 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a14c4f785f4514ae74b8a0b3073503" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.840359 5106 tablet_bootstrap.cc:654] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.843510 4608 leader_election.cc:302] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.846949 5103 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:34.848495 5990 consensus_meta.cc:287] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.670s user 0.004s sys 0.000s
I20240503 02:12:34.848712 5990 raft_consensus.cc:504] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.849290 4808 raft_consensus.cc:2379] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:34.849720 5990 leader_election.cc:288] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:34.850678 5103 tablet_bootstrap.cc:654] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.850841 6024 consensus_meta.cc:287] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.512s user 0.004s sys 0.000s
I20240503 02:12:34.850924 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:34.851034 6024 raft_consensus.cc:504] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.851212 4670 raft_consensus.cc:2379] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:34.851835 6024 leader_election.cc:288] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.852560 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 541) took 2626 ms. Trace:
I20240503 02:12:34.852722 5115 rpcz_store.cc:276] 0503 02:12:32.225667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.228448 (+ 2781us) service_pool.cc:225] Handling call
0503 02:12:34.204536 (+1976088us) ts_tablet_manager.cc:1120] Deleting tablet 34eb69d5f4bd427a8e430a00fb076f83
0503 02:12:34.204545 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.321503 (+116958us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.567789 (+246286us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.852470 (+284681us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967062}
I20240503 02:12:34.853473 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 544) took 2627 ms. Trace:
I20240503 02:12:34.853592 5115 rpcz_store.cc:276] 0503 02:12:32.226056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.227295 (+ 1239us) service_pool.cc:225] Handling call
0503 02:12:34.853191 (+2625896us) ts_tablet_manager.cc:1120] Deleting tablet fdc2850c07d042a7951a5fbeb0e4945d
0503 02:12:34.853205 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.853412 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2614189}
I20240503 02:12:34.854071 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 546) took 2307 ms. Trace:
I20240503 02:12:34.854199 5115 rpcz_store.cc:276] 0503 02:12:32.548056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.548250 (+ 194us) service_pool.cc:225] Handling call
0503 02:12:34.853888 (+2305638us) ts_tablet_manager.cc:1120] Deleting tablet ec641e403ed843fda31dabd3920aeb68
0503 02:12:34.853900 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.853995 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2305311}
I20240503 02:12:34.854704 5106 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.854941 5106 ts_tablet_manager.cc:1367] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240503 02:12:34.856232 5106 raft_consensus.cc:348] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.856550 5106 raft_consensus.cc:374] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.856694 5106 raft_consensus.cc:729] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.857007 5106 consensus_queue.cc:260] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.857985 5106 ts_tablet_manager.cc:1398] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.858461 5106 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:34.859946 4777 consensus_meta.cc:287] T 6ecdb634fb1c4337b2eae0c844fbb1dc P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.750s user 0.005s sys 0.000s
I20240503 02:12:34.862967 5106 tablet_bootstrap.cc:654] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.863889 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22776c81d1a24d368abb68274cf32a0b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:34.864270 4525 raft_consensus.cc:2379] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acc6d33969e74c209310005ae911c405 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:34.866616 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.867715 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.868140 4525 raft_consensus.cc:2454] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.868882 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_487d9f1f0e804199b88dcdc8c9f6d0d8" } new_table_name: "default.Table_2393371f385c4869b2be72bd2dad35bf" modify_external_catalogs: true
I20240503 02:12:34.868940 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f425f7b64daf435a8f9367ef0c1dbae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.869756 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f425f7b64daf435a8f9367ef0c1dbae1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.854679 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 547) took 2302 ms. Trace:
I20240503 02:12:34.872258 5115 rpcz_store.cc:276] 0503 02:12:32.553735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.554088 (+ 353us) service_pool.cc:225] Handling call
0503 02:12:34.854487 (+2300399us) ts_tablet_manager.cc:1120] Deleting tablet fcfb174825f1441f861ae6816137f855
0503 02:12:34.854498 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.854600 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2300118}
I20240503 02:12:34.872992 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 549) took 2153 ms. Trace:
I20240503 02:12:34.873279 5103 tablet_bootstrap.cc:492] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.873139 5115 rpcz_store.cc:276] 0503 02:12:32.719683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.719741 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:34.872702 (+2152961us) ts_tablet_manager.cc:1120] Deleting tablet 17a24b5c83b649e79644b08253d851d6
0503 02:12:34.872720 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.872896 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2152661}
I20240503 02:12:34.873503 5103 ts_tablet_manager.cc:1367] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240503 02:12:34.873638 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 550) took 2153 ms. Trace:
I20240503 02:12:34.873768 5115 rpcz_store.cc:276] 0503 02:12:32.720188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.720229 (+ 41us) service_pool.cc:225] Handling call
0503 02:12:34.873453 (+2153224us) ts_tablet_manager.cc:1120] Deleting tablet 552dad16f30a410280c87a6c5ef0eaa4
0503 02:12:34.873465 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.873573 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153008}
I20240503 02:12:34.874190 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 551) took 2088 ms. Trace:
I20240503 02:12:34.874595 5103 raft_consensus.cc:348] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.874943 5103 raft_consensus.cc:374] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.874274 5115 rpcz_store.cc:276] 0503 02:12:32.785269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.785322 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:34.874028 (+2088706us) ts_tablet_manager.cc:1120] Deleting tablet 50a43e428b804729adf9d8a3133c90ff
0503 02:12:34.874038 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.874134 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088440}
I20240503 02:12:34.875030 5103 raft_consensus.cc:729] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.875283 5115 tablet_replica.cc:330] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:34.875416 5103 consensus_queue.cc:260] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.875582 5115 raft_consensus.cc:2229] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:34.875871 5115 raft_consensus.cc:2258] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.876672 5115 ts_tablet_manager.cc:1875] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:34.877207 5103 ts_tablet_manager.cc:1398] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:34.877823 5103 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.879143 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:34.880049 4519 raft_consensus.cc:2454] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:34.880153 5103 tablet_bootstrap.cc:654] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:34.881915 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c6c80efe35343989f46a2c181f38129 consensus state has no leader
I20240503 02:12:34.883352 5106 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.883589 5106 ts_tablet_manager.cc:1367] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.005s
I20240503 02:12:34.884572 5106 raft_consensus.cc:348] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.885022 5106 raft_consensus.cc:374] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.885157 5106 raft_consensus.cc:729] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.885329 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 522) took 6033 ms. Trace:
I20240503 02:12:34.885530 5106 consensus_queue.cc:260] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.885579 5114 rpcz_store.cc:276] 0503 02:12:28.851601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.524307 (+2672706us) service_pool.cc:225] Handling call
0503 02:12:34.481158 (+2956851us) ts_tablet_manager.cc:1120] Deleting tablet 4064ff59de474e0db3a578eae10c3fa6
0503 02:12:34.481171 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.564586 (+ 83415us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.709669 (+145083us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.885300 (+175631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2956677}
I20240503 02:12:34.886471 5106 ts_tablet_manager.cc:1398] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:34.886891 5106 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.889190 5106 tablet_bootstrap.cc:654] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.893926 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 4064ff59de474e0db3a578eae10c3fa6 (table default.Table_77eae5a54c4c440a9928ae87553755be [id=5108d90cc915403ba879db13f6a9f1be]) successfully deleted
I20240503 02:12:34.903117 5103 tablet_bootstrap.cc:492] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.903285 5103 ts_tablet_manager.cc:1367] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 02:12:34.906313 5103 raft_consensus.cc:348] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.906560 5103 raft_consensus.cc:374] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.906638 5103 raft_consensus.cc:729] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.906898 5103 consensus_queue.cc:260] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.907003 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 523) took 6055 ms. Trace:
I20240503 02:12:34.912801 4746 leader_election.cc:302] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.913148 5990 raft_consensus.cc:2726] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.913328 4746 leader_election.cc:302] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.913704 6024 raft_consensus.cc:2781] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.913753 5991 raft_consensus.cc:2781] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.913921 5991 raft_consensus.cc:686] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:34.914198 5991 consensus_queue.cc:237] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.913851 6024 raft_consensus.cc:482] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.914640 6024 raft_consensus.cc:3037] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.914921 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d693ea577510429c96e04408da3828e1" } new_table_name: "default.Table_2d181d60cede45628a69e42c770848f9" modify_external_catalogs: true
W20240503 02:12:34.915917 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 72511.2us
I20240503 02:12:34.916316 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 547) took 3374 ms. Trace:
I20240503 02:12:34.907123 5114 rpcz_store.cc:276] 0503 02:12:28.851694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.524492 (+2672798us) service_pool.cc:225] Handling call
0503 02:12:34.906860 (+3382368us) ts_tablet_manager.cc:1120] Deleting tablet 0cc2ebfb11aa4840adcc35da73be75a2
0503 02:12:34.906871 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.906984 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3382199}
I20240503 02:12:34.916432 5159 rpcz_store.cc:276] 0503 02:12:31.537210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.537332 (+ 122us) service_pool.cc:225] Handling call
0503 02:12:34.291416 (+2754084us) ts_tablet_manager.cc:1120] Deleting tablet d271d2c112ba44b68d2933a6ebfe7a18
0503 02:12:34.291428 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.575115 (+283687us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x616000142f54. stack: 00007fced75a0756 00007fced75a0510 00007fced65d348d 0000000000955804 00000000009557cd 000000000095519a 00007fcee2e0ea00 00007fcee40e8b83 00007fcee6b7c57c 00007fcee6bae8d4 00007fcee6b8fbc0 00007fcee6b8f94d 0000000000a60bcc 00007fced75e6b74 00007fced75ea19b 00007fced75ea018
0503 02:12:34.700323 (+125208us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.753502 (+ 53179us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:34.911462 (+157960us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10584,"spinlock_wait_cycles":650931456,"tablet-delete.queue_time_us":2733254}
W20240503 02:12:34.918449 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab093285c39b4ce7a0bb3de9d016a918 consensus state has no leader
W20240503 02:12:34.919170 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f4aad131b884c54b69b3d1d9420662c consensus state has no leader
I20240503 02:12:34.920616 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_acdeb2544a5243f184185b6c97b607f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.921195 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acdeb2544a5243f184185b6c97b607f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.922777 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_28c626ea06e84dc98ac97225be580b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.923307 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28c626ea06e84dc98ac97225be580b7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.923890 6057 raft_consensus.cc:482] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.925230 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_44989b3752354e3e96f57a2f15a27ec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.925757 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44989b3752354e3e96f57a2f15a27ec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.927762 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 548) took 3380 ms. Trace:
I20240503 02:12:34.927712 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.928037 4808 raft_consensus.cc:2454] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:34.927920 5159 rpcz_store.cc:276] 0503 02:12:31.537369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.537613 (+ 244us) service_pool.cc:225] Handling call
0503 02:12:34.916744 (+3379131us) ts_tablet_manager.cc:1120] Deleting tablet fdc2850c07d042a7951a5fbeb0e4945d
0503 02:12:34.916754 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.917631 (+ 877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3358513}
I20240503 02:12:34.929328 5106 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:34.929502 5106 ts_tablet_manager.cc:1367] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.006s
I20240503 02:12:34.930112 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet d271d2c112ba44b68d2933a6ebfe7a18 (table default.Table_d54df163e4634a779d3bcfddb083db60 [id=363231755ce84709b926a7416f3bc40a]) successfully deleted
W20240503 02:12:34.930548 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fdc2850c07d042a7951a5fbeb0e4945d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc2850c07d042a7951a5fbeb0e4945d already in progress: creating tablet
I20240503 02:12:34.930506 5106 raft_consensus.cc:348] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.930711 5106 raft_consensus.cc:374] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.930802 5106 raft_consensus.cc:729] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.931103 5106 consensus_queue.cc:260] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.931787 5106 ts_tablet_manager.cc:1398] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:34.932401 5106 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:34.933765 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 549) took 3390 ms. Trace:
I20240503 02:12:34.934160 6057 raft_consensus.cc:504] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.935545 5828 raft_consensus.cc:482] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.935681 5828 raft_consensus.cc:504] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.935887 5106 tablet_bootstrap.cc:654] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.918361 5103 ts_tablet_manager.cc:1398] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:12:34.936823 5103 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.938670 5103 tablet_bootstrap.cc:654] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.944168 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_612f648735a34eff8dafc63f119da941" } modify_external_catalogs: true
I20240503 02:12:34.944296 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8b93d7e7be90441ea960cdf3b81e4fc7" } modify_external_catalogs: true
I20240503 02:12:34.944415 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_612f648735a34eff8dafc63f119da941" } modify_external_catalogs: true
I20240503 02:12:34.944518 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8b93d7e7be90441ea960cdf3b81e4fc7" } modify_external_catalogs: true
I20240503 02:12:34.945547 6013 raft_consensus.cc:482] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.945715 6013 raft_consensus.cc:504] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.946578 6013 leader_election.cc:288] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:34.947072 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.947137 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fe7f91fccadf4fa482c48523fb77a990" } modify_external_catalogs: true
I20240503 02:12:34.947433 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fe7f91fccadf4fa482c48523fb77a990" } modify_external_catalogs: true
I20240503 02:12:34.949051 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8750b02c5936412db525d2d3e77befeb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.949287 4811 raft_consensus.cc:2379] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:34.949347 6057 leader_election.cc:288] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.952337 5828 leader_election.cc:288] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.954754 6075 raft_consensus.cc:504] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.955562 6075 leader_election.cc:288] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.955718 5828 raft_consensus.cc:482] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.955842 5828 raft_consensus.cc:504] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.956568 5828 leader_election.cc:288] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.957710 4608 leader_election.cc:302] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:34.958127 4608 leader_election.cc:302] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:34.958328 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.958400 6107 raft_consensus.cc:2781] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.958546 6107 raft_consensus.cc:482] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.958650 6107 raft_consensus.cc:3037] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.958586 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.959264 4608 leader_election.cc:302] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.959534 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.960233 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.960523 4525 raft_consensus.cc:2454] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.961459 6075 raft_consensus.cc:2781] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.961704 6075 raft_consensus.cc:686] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:34.962082 6075 consensus_queue.cc:237] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:34.962951 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.963913 5105 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:34.964393 4746 leader_election.cc:341] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.964568 4746 leader_election.cc:302] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.964779 5828 raft_consensus.cc:2781] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.964872 5828 raft_consensus.cc:482] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.964957 5828 raft_consensus.cc:3037] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:34.964794 4746 leader_election.cc:341] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:34.965608 4746 leader_election.cc:341] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:34.965938 4746 leader_election.cc:341] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.966032 4746 leader_election.cc:302] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.966367 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:34.966533 6057 raft_consensus.cc:2726] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.966547 4811 raft_consensus.cc:2454] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:34.956842 5159 rpcz_store.cc:276] 0503 02:12:31.537545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.537877 (+ 332us) service_pool.cc:225] Handling call
0503 02:12:34.928273 (+3390396us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:34.928284 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.928399 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369983}
I20240503 02:12:34.967180 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 550) took 2522 ms. Trace:
I20240503 02:12:34.967253 5159 rpcz_store.cc:276] 0503 02:12:32.445246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.445315 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:34.967069 (+2521754us) ts_tablet_manager.cc:1120] Deleting tablet ec641e403ed843fda31dabd3920aeb68
0503 02:12:34.967078 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.967166 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2521483}
W20240503 02:12:34.967307 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0cc2ebfb11aa4840adcc35da73be75a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc2ebfb11aa4840adcc35da73be75a2 already in progress: creating tablet
I20240503 02:12:34.967532 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 551) took 2498 ms. Trace:
I20240503 02:12:34.967589 5159 rpcz_store.cc:276] 0503 02:12:32.469584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.469658 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:34.967434 (+2497776us) ts_tablet_manager.cc:1120] Deleting tablet fcfb174825f1441f861ae6816137f855
0503 02:12:34.967452 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.967520 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2497505}
I20240503 02:12:34.968300 4462 leader_election.cc:302] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:34.968248 5105 ts_tablet_manager.cc:1367] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.140s user 0.024s sys 0.028s
I20240503 02:12:34.968585 6013 raft_consensus.cc:2781] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.968683 6013 raft_consensus.cc:482] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.968778 6013 raft_consensus.cc:3037] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.969166 5105 raft_consensus.cc:348] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.969354 5105 raft_consensus.cc:374] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.969412 5105 raft_consensus.cc:729] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.969421 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_074e9d0fdabc4c3d9e642fb8fefce869"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.969513 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1a5f115728484ec18350f73f8ffa672b" } modify_external_catalogs: true
I20240503 02:12:34.969614 5105 consensus_queue.cc:260] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.969744 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1a5f115728484ec18350f73f8ffa672b" } modify_external_catalogs: true
W20240503 02:12:34.970016 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_074e9d0fdabc4c3d9e642fb8fefce869 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.970153 5105 ts_tablet_manager.cc:1398] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:34.971554 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d64beb8ee02743139edc02931b1f97ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:34.972090 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d64beb8ee02743139edc02931b1f97ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:34.973580 5159 tablet_replica.cc:330] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:34.973889 5159 raft_consensus.cc:2229] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:34.974193 5159 raft_consensus.cc:2258] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.974753 5159 ts_tablet_manager.cc:1875] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:34.936798 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 533) took 4736 ms. Trace:
I20240503 02:12:34.976032 5114 rpcz_store.cc:276] 0503 02:12:30.200244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.026485 (+1826241us) service_pool.cc:225] Handling call
0503 02:12:34.936608 (+2910123us) ts_tablet_manager.cc:1120] Deleting tablet 3478033c4ed94268a2436a2942c27a68
0503 02:12:34.936616 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:34.936731 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2909876}
I20240503 02:12:34.976418 5114 tablet_replica.cc:330] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:34.976645 5114 raft_consensus.cc:2229] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:34.976867 5114 raft_consensus.cc:2258] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:34.977497 5114 ts_tablet_manager.cc:1875] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:34.978435 5105 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:34.978552 6057 raft_consensus.cc:482] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.978672 6057 raft_consensus.cc:504] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.979279 6057 leader_election.cc:288] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:34.980033 5105 tablet_bootstrap.cc:654] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.980034 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d919b9262ac48cbb8b3f893918b5590" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:34.980135 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d919b9262ac48cbb8b3f893918b5590" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:34.980443 4668 raft_consensus.cc:2454] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:34.980938 4746 leader_election.cc:302] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:34.981281 4746 leader_election.cc:341] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.981475 6057 raft_consensus.cc:2781] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.981851 6057 raft_consensus.cc:482] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.981932 6057 raft_consensus.cc:3037] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:34.991911 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3478033c4ed94268a2436a2942c27a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3478033c4ed94268a2436a2942c27a68 already in progress: creating tablet
I20240503 02:12:34.992033 5103 tablet_bootstrap.cc:492] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:34.992182 5103 ts_tablet_manager.cc:1367] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.002s
W20240503 02:12:34.992233 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 48e713ee37d042309166b0f118e14986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e713ee37d042309166b0f118e14986 already in progress: creating tablet
W20240503 02:12:34.992614 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fcfb174825f1441f861ae6816137f855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcfb174825f1441f861ae6816137f855 already in progress: creating tablet
I20240503 02:12:34.992995 5103 raft_consensus.cc:348] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.993214 5103 raft_consensus.cc:374] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.993288 5103 raft_consensus.cc:729] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.993484 4608 leader_election.cc:302] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:34.993528 5103 consensus_queue.cc:260] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:34.993747 5999 raft_consensus.cc:2726] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:34.993935 5999 raft_consensus.cc:3037] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.994105 5103 ts_tablet_manager.cc:1398] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:34.994436 5103 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:34.996299 5103 tablet_bootstrap.cc:654] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.000998 5868 consensus_queue.cc:1035] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.018349 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet b8474c841cb64373a223ad76b43ecf5f with delete_type TABLET_DATA_DELETED (replaced by tablet 73198ec769a9486da3f85e8fcbef8e78) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.019542 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.021689 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f6279f16dc42d69e794e3c6586163d with delete_type TABLET_DATA_DELETED (replaced by tablet f50322e5e3eb4eb3a1b3479fd6c992ea) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.023065 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d555c8d8844ee49c960a5279485480 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.023716 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 8135907b78284d7ca74396cc77dd0612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.029073 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447242714c844e685faca5aed71a7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.029675 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet cef84b4e6ba24150905994d45ffd0b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.030938 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.030938 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.031157 4810 raft_consensus.cc:3037] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:35.035218 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43fc277fa4884c7f92118e333226e800 consensus state has no leader
I20240503 02:12:35.037675 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a9135679f7fb423cb1d8d69c3430feb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.037796 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:35.038313 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9135679f7fb423cb1d8d69c3430feb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.038467 6079 raft_consensus.cc:504] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.039072 6079 leader_election.cc:288] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.039506 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 34eb69d5f4bd427a8e430a00fb076f83 (table default.Table_a6d935ff1f554a62a312091e6252d15a [id=7118c326733f4c6695909797bd035f9e]) successfully deleted
W20240503 02:12:35.039804 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet fdc2850c07d042a7951a5fbeb0e4945d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc2850c07d042a7951a5fbeb0e4945d already in progress: creating tablet
W20240503 02:12:35.040066 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet fcfb174825f1441f861ae6816137f855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcfb174825f1441f861ae6816137f855 already in progress: creating tablet
W20240503 02:12:35.040243 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 17a24b5c83b649e79644b08253d851d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a24b5c83b649e79644b08253d851d6 already in progress: creating tablet
W20240503 02:12:35.040397 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 552dad16f30a410280c87a6c5ef0eaa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552dad16f30a410280c87a6c5ef0eaa4 already in progress: creating tablet
W20240503 02:12:35.040760 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 50a43e428b804729adf9d8a3133c90ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50a43e428b804729adf9d8a3133c90ff already in progress: creating tablet
I20240503 02:12:35.042464 6130 tablet.cc:1722] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 02:12:35.044108 5851 consensus_meta.cc:287] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.613s user 0.002s sys 0.001s
I20240503 02:12:35.044185 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.044261 5851 raft_consensus.cc:504] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.044391 4672 raft_consensus.cc:3037] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.045023 5851 leader_election.cc:288] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.045405 6013 raft_consensus.cc:504] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.046875 6013 leader_election.cc:288] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:35.047741 4670 raft_consensus.cc:2379] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:35.047991 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.048720 4675 raft_consensus.cc:2379] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:35.049926 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2ea009902d854d4096d91c7e2d54a289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.050565 4529 raft_consensus.cc:3037] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:35.050572 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ea009902d854d4096d91c7e2d54a289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.051697 5159 ts_tablet_manager.cc:1888] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:35.051816 5159 log.cc:1196] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b080fc5ffc564420a0b7f2b7460ebbbb
I20240503 02:12:35.052181 5159 ts_tablet_manager.cc:1909] T b080fc5ffc564420a0b7f2b7460ebbbb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:35.052088 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_88f4791ae6054ec5becf0d13aec8be47" } modify_external_catalogs: true
I20240503 02:12:35.052309 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_88f4791ae6054ec5becf0d13aec8be47" } modify_external_catalogs: true
I20240503 02:12:35.053213 6001 raft_consensus.cc:482] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.053804 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.054003 4807 raft_consensus.cc:3037] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.053427 6001 raft_consensus.cc:504] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.054764 6001 leader_election.cc:288] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.055565 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.055639 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.055757 4811 raft_consensus.cc:3037] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.056258 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:35.056716 4746 leader_election.cc:341] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:35.061571 4746 leader_election.cc:341] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.061765 4746 leader_election.cc:302] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.062054 6079 raft_consensus.cc:2726] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.069656 6079 raft_consensus.cc:482] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.069849 6079 raft_consensus.cc:504] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.070528 6079 leader_election.cc:288] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.071301 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:35.072196 4746 leader_election.cc:341] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.072542 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.072609 5106 tablet_bootstrap.cc:492] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.072825 5106 ts_tablet_manager.cc:1367] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.002s
W20240503 02:12:35.073204 4746 leader_election.cc:341] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.073309 4746 leader_election.cc:302] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.073743 6079 raft_consensus.cc:2726] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.074193 5106 raft_consensus.cc:348] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.074559 5106 raft_consensus.cc:374] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.074661 5106 raft_consensus.cc:729] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.074962 5106 consensus_queue.cc:260] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.075716 5106 ts_tablet_manager.cc:1398] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:35.076174 5106 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.079075 5106 tablet_bootstrap.cc:654] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.079411 6135 tablet.cc:1722] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:35.084444 5937 webserver.cc:460] Webserver started at http://127.4.17.124:46343/ using document root <none> and password file <none>
I20240503 02:12:35.090925 6041 consensus_queue.cc:1035] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.093734 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 17afd88bf6c040ad9e8d571eccd348dc
I20240503 02:12:35.094175 5937 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:35.094264 4266 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f50322e5e3eb4eb3a1b3479fd6c992ea
I20240503 02:12:35.094594 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f99fb87942724549bded0fb6c29f3420" } new_table_name: "default.Table_bb63ba232a074c0692fa657db43e25dc" modify_external_catalogs: true
I20240503 02:12:35.094707 5937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:35.094642 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5956db05373844bcafa9952fa9f6e513" } new_table_name: "default.Table_80dedb822df54ccc9e63c27319fec3b9" modify_external_catalogs: true
I20240503 02:12:35.097710 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:35.098883 5105 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.098862 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.099046 5105 ts_tablet_manager.cc:1367] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.121s user 0.037s sys 0.027s
I20240503 02:12:35.099707 5106 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.099879 5106 ts_tablet_manager.cc:1367] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.002s
I20240503 02:12:35.099802 5105 raft_consensus.cc:348] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.100003 5105 raft_consensus.cc:374] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.100064 5105 raft_consensus.cc:729] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.100270 5105 consensus_queue.cc:260] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.100976 5105 ts_tablet_manager.cc:1398] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:35.101292 5105 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.102123 5106 raft_consensus.cc:348] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.102483 5106 raft_consensus.cc:374] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.102594 5106 raft_consensus.cc:729] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.102816 5106 consensus_queue.cc:260] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.104094 5106 ts_tablet_manager.cc:1398] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:35.104468 5106 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:35.105082 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 51893.5us
I20240503 02:12:35.105300 5105 tablet_bootstrap.cc:654] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.105895 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.107009 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053537029a44d5a519b612d09d19a1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.107308 4519 raft_consensus.cc:2379] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:35.107429 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:35.108392 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d58f97b58481ea1dfa75397a32ae2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.108668 4519 raft_consensus.cc:2379] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:35.110416 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:35.110751 4608 leader_election.cc:341] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:35.110991 4608 leader_election.cc:341] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.110868 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_366708ec2da84f23933deff6182b9fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.111429 4608 leader_election.cc:302] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.111724 6075 raft_consensus.cc:2726] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 02:12:35.111743 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_366708ec2da84f23933deff6182b9fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.112227 4608 leader_election.cc:302] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.114557 6001 raft_consensus.cc:3037] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.115109 5106 tablet_bootstrap.cc:654] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.115886 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.121850 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 7d919b9262ac48cbb8b3f893918b5590 (table default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95]). Expected version 3 got 0
I20240503 02:12:35.122184 5105 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.122172 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:35.122318 5105 ts_tablet_manager.cc:1367] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20240503 02:12:35.123072 5105 raft_consensus.cc:348] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.123275 5105 raft_consensus.cc:374] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.123332 5105 raft_consensus.cc:729] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.123533 5105 consensus_queue.cc:260] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.124018 5105 ts_tablet_manager.cc:1398] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:35.124296 5105 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.125689 5105 tablet_bootstrap.cc:654] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.127468 6140 tablet.cc:1722] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:35.128399 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b (DEFAULT_TABLE table=default.Table_5ad66186bf0948d996f2098e51b3a661 [id=43ff2e875e4e480a871ca527675be18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.128823 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_94e778467d034214b379477ebfe9694e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.129467 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94e778467d034214b379477ebfe9694e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.131197 4672 raft_consensus.cc:2454] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.131529 5114 ts_tablet_manager.cc:1888] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:35.131503 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5a2971f0db2147bb875eba0e1a871f95" } new_table_name: "default.Table_7d5419d4bbf34b67bcc05cf32663c2df" modify_external_catalogs: true
I20240503 02:12:35.131659 5114 log.cc:1196] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/dedead5a2bf347f6991cce8e9b4db708
I20240503 02:12:35.132025 5114 ts_tablet_manager.cc:1909] T dedead5a2bf347f6991cce8e9b4db708 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:35.134536 5828 raft_consensus.cc:504] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.135293 5828 leader_election.cc:288] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.137215 6136 consensus_queue.cc:1035] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.137799 5105 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.137933 5105 ts_tablet_manager.cc:1367] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 02:12:35.138701 5105 raft_consensus.cc:348] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.138881 5105 raft_consensus.cc:374] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.138935 5105 raft_consensus.cc:729] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.139132 5105 consensus_queue.cc:260] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.139714 5105 ts_tablet_manager.cc:1398] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:35.145403 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.146171 4746 leader_election.cc:302] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.147310 5828 raft_consensus.cc:2781] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.147433 5828 raft_consensus.cc:686] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:35.147748 5828 consensus_queue.cc:237] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.151404 4672 raft_consensus.cc:3037] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.158691 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53017352b8f34955b116bbfd32f04854" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.159986 4525 raft_consensus.cc:3037] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.161381 6136 consensus_queue.cc:1035] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.163170 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 531) took 5381 ms. Trace:
I20240503 02:12:35.163339 5114 rpcz_store.cc:276] 0503 02:12:29.781898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.653756 (+1871858us) service_pool.cc:225] Handling call
0503 02:12:34.976343 (+3322587us) ts_tablet_manager.cc:1120] Deleting tablet dedead5a2bf347f6991cce8e9b4db708
0503 02:12:34.976353 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.074605 (+ 98252us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.131387 (+ 56782us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.163107 (+ 31720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2937407}
I20240503 02:12:35.163756 5114 tablet_replica.cc:330] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:35.163992 5114 raft_consensus.cc:2229] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:35.164201 5114 raft_consensus.cc:2258] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.164820 5114 ts_tablet_manager.cc:1875] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.170861 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet dedead5a2bf347f6991cce8e9b4db708 (table default.Table_873e48ca8c434f55927075d87ac065b1 [id=01ac062e9b344819ad99b55ae572916d]) successfully deleted
I20240503 02:12:35.172669 5105 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.174643 5105 tablet_bootstrap.cc:654] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.175366 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.179169 5828 raft_consensus.cc:482] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.179328 5828 raft_consensus.cc:504] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.180071 5828 leader_election.cc:288] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.181080 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.181170 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.181500 5105 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.181627 5105 ts_tablet_manager.cc:1367] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
W20240503 02:12:35.181896 4746 leader_election.cc:341] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:35.182134 4746 leader_election.cc:341] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.182224 4746 leader_election.cc:302] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.182497 5105 raft_consensus.cc:348] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.182693 5105 raft_consensus.cc:374] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.182706 5828 raft_consensus.cc:2726] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.182742 5105 raft_consensus.cc:729] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.183022 5105 consensus_queue.cc:260] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.183738 5105 ts_tablet_manager.cc:1398] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.183997 5105 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.192565 5105 tablet_bootstrap.cc:654] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.195662 6136 consensus_queue.cc:1035] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.195297 6075 consensus_queue.cc:1035] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.208945 5105 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.209108 5105 ts_tablet_manager.cc:1367] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20240503 02:12:35.211935 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_68c5b84eae2d40739236f3f03dc38fdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.212256 5105 raft_consensus.cc:348] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.212432 5105 raft_consensus.cc:374] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.212482 5105 raft_consensus.cc:729] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:35.212672 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c5b84eae2d40739236f3f03dc38fdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.212724 5105 consensus_queue.cc:260] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:35.212852 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7178760c21904207b6c8317093871f63 consensus state has no leader
I20240503 02:12:35.213138 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6ecdb634fb1c4337b2eae0c844fbb1dc
I20240503 02:12:35.213313 5105 ts_tablet_manager.cc:1398] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:35.213277 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_074e9d0fdabc4c3d9e642fb8fefce869" } new_table_name: "default.Table_72c760ead308416095fdc381a2737426" modify_external_catalogs: true
I20240503 02:12:35.213624 5105 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.213879 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ee1fac803dd84eee8722f0a14559faf6
W20240503 02:12:35.214092 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b32c377006a24372859ed44d59cf90ee consensus state has no leader
I20240503 02:12:35.215282 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9290a61c21fa456eb3c4f9b0dbebc864
I20240503 02:12:35.215348 5105 tablet_bootstrap.cc:654] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:35.215283 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4225c1bc133d4e18a4a47fd58820ca10 consensus state has no leader
I20240503 02:12:35.225391 5999 raft_consensus.cc:2726] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.229794 5114 ts_tablet_manager.cc:1888] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:35.229974 5114 log.cc:1196] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/9327a9e5472b4eda854320c3a95f8e56
W20240503 02:12:35.230913 6078 consensus_meta.cc:287] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.713s user 0.004s sys 0.000s
I20240503 02:12:35.231079 5114 ts_tablet_manager.cc:1909] T 9327a9e5472b4eda854320c3a95f8e56 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:35.231053 6078 raft_consensus.cc:504] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.232411 6078 leader_election.cc:288] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.235771 6013 consensus_queue.cc:1035] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.241640 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3295e34cbed747c68141113383a688b1" } new_table_name: "default.Table_453e312025324be6a4d15ae5116a46fa" modify_external_catalogs: true
I20240503 02:12:35.242142 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cab73492219d41c4bf3fafcf6ac31309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.242857 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cab73492219d41c4bf3fafcf6ac31309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.242717 4266 catalog_manager.cc:5484] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:35.243238 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:35.243587 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet ea4dac17dca3442d8934ac357779b78a (table default.Table_55c37198f9d2480784f643011b70dee8 [id=89060d0da21a4cf8b3d3c1ab1a8a3749]). Expected version 2 got 0
I20240503 02:12:35.244064 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7d919b9262ac48cbb8b3f893918b5590 (table default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95]). Expected version 3 got 0
I20240503 02:12:35.245481 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19ed19da5b74583bfc3b21d88e937c9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.245699 4665 raft_consensus.cc:3037] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.245857 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19ed19da5b74583bfc3b21d88e937c9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.246066 4529 raft_consensus.cc:3037] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.249408 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.250110 4525 raft_consensus.cc:2454] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.250746 4746 leader_election.cc:302] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:35.251224 6078 raft_consensus.cc:2781] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.251346 6078 raft_consensus.cc:686] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:35.251639 6078 consensus_queue.cc:237] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.263300 5105 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.263478 5105 ts_tablet_manager.cc:1367] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.050s user 0.026s sys 0.021s
I20240503 02:12:35.264452 5105 raft_consensus.cc:348] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:35.264657 5105 raft_consensus.cc:374] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.264715 5105 raft_consensus.cc:729] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.264909 5105 consensus_queue.cc:260] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:35.265470 5105 ts_tablet_manager.cc:1398] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.268723 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.269908 5105 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.273025 5828 consensus_queue.cc:1035] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.277261 5105 tablet_bootstrap.cc:654] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.282429 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 539) took 4308 ms. Trace:
I20240503 02:12:35.282598 5114 rpcz_store.cc:276] 0503 02:12:30.974257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.069853 (+1095596us) service_pool.cc:225] Handling call
0503 02:12:35.163680 (+3093827us) ts_tablet_manager.cc:1120] Deleting tablet 9327a9e5472b4eda854320c3a95f8e56
0503 02:12:35.163689 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.225763 (+ 62074us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.228785 (+ 3022us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.282373 (+ 53588us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1181056,"tablet-delete.queue_time_us":3093580}
I20240503 02:12:35.282979 5114 tablet_replica.cc:330] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:35.283217 5114 raft_consensus.cc:2229] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:35.283407 5114 raft_consensus.cc:2258] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.284014 5114 ts_tablet_manager.cc:1875] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.284665 5103 tablet_bootstrap.cc:492] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:35.284819 5103 ts_tablet_manager.cc:1367] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.290s user 0.017s sys 0.000s
I20240503 02:12:35.285650 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:35.286531 6041 consensus_queue.cc:1035] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.288079 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1fac803dd84eee8722f0a14559faf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.289124 5105 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.289269 5105 ts_tablet_manager.cc:1367] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 02:12:35.290871 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1fac803dd84eee8722f0a14559faf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.291802 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1fac803dd84eee8722f0a14559faf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:35.293097 5105 raft_consensus.cc:348] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.293309 5105 raft_consensus.cc:374] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.293371 5105 raft_consensus.cc:729] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.293576 5105 consensus_queue.cc:260] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.294117 5105 ts_tablet_manager.cc:1398] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:12:35.294411 5105 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.295126 5103 raft_consensus.cc:348] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.299257 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.299690 5105 tablet_bootstrap.cc:654] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.299762 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_55c37198f9d2480784f643011b70dee8" } modify_external_catalogs: true
I20240503 02:12:35.299840 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3fe44eb302634e1a8900af7fba21d5b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.301213 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9f8e5dd83dd347d29e90e271ed77b18a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.301961 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_55c37198f9d2480784f643011b70dee8" } modify_external_catalogs: true
I20240503 02:12:35.302686 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d0e9b6b6436b4eb593d72514890ee977" } modify_external_catalogs: true
I20240503 02:12:35.302938 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:35.302930 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d0e9b6b6436b4eb593d72514890ee977" } modify_external_catalogs: true
I20240503 02:12:35.305053 5828 raft_consensus.cc:482] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.305230 5828 raft_consensus.cc:504] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.306272 5828 leader_election.cc:288] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.306831 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:35.307027 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fe44eb302634e1a8900af7fba21d5b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:35.307543 4746 leader_election.cc:341] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.309095 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b (DEFAULT_TABLE table=default.Table_5ad66186bf0948d996f2098e51b3a661 [id=43ff2e875e4e480a871ca527675be18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.309628 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:35.310773 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f8e5dd83dd347d29e90e271ed77b18a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.328742 6057 raft_consensus.cc:504] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.329857 6057 leader_election.cc:288] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.330577 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 553) took 2681 ms. Trace:
I20240503 02:12:35.330705 5159 rpcz_store.cc:276] 0503 02:12:32.649431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.649513 (+ 82us) service_pool.cc:225] Handling call
0503 02:12:34.967816 (+2318303us) ts_tablet_manager.cc:1120] Deleting tablet b080fc5ffc564420a0b7f2b7460ebbbb
0503 02:12:34.967823 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.038610 (+ 70787us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.051555 (+ 12945us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.330542 (+278987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2318071}
I20240503 02:12:35.331317 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 554) took 2681 ms. Trace:
I20240503 02:12:35.331399 5159 rpcz_store.cc:276] 0503 02:12:32.649787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.649835 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:35.331193 (+2681358us) ts_tablet_manager.cc:1120] Deleting tablet c0b9003f248b47a9952e6f995887ae0c
0503 02:12:35.331208 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.331297 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681133}
I20240503 02:12:35.331679 5159 tablet_replica.cc:330] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:35.331971 5159 raft_consensus.cc:2229] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:35.332208 5159 raft_consensus.cc:2258] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.333024 5159 ts_tablet_manager.cc:1875] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:35.337149 4673 consensus_meta.cc:287] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.666s user 0.002s sys 0.002s
I20240503 02:12:35.343613 4673 raft_consensus.cc:2454] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
W20240503 02:12:35.340039 4642 consensus_meta.cc:287] T 3ee32345fb9349f19af09a3447c4caad P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.659s user 0.000s sys 0.001s
W20240503 02:12:35.343411 4647 consensus_meta.cc:287] T a8ecddfcfed2436e98bb7b6b38beb03f P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.636s user 0.000s sys 0.000s
I20240503 02:12:35.351155 6057 raft_consensus.cc:482] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.351414 5999 raft_consensus.cc:482] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.351373 6057 raft_consensus.cc:504] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.351526 5999 raft_consensus.cc:504] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.352070 5999 leader_election.cc:288] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.352058 6057 leader_election.cc:288] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.353690 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:35.353899 4805 raft_consensus.cc:2454] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:35.353857 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d919b9262ac48cbb8b3f893918b5590" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.354014 4673 raft_consensus.cc:3037] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.354523 4608 leader_election.cc:302] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:35.355841 5999 raft_consensus.cc:2781] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.355826 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.355933 5999 raft_consensus.cc:482] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.355991 5999 raft_consensus.cc:3037] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.357308 6057 raft_consensus.cc:482] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.357429 6057 raft_consensus.cc:504] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.358139 6057 leader_election.cc:288] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:35.358570 4746 leader_election.cc:341] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.359910 5115 ts_tablet_manager.cc:1888] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:35.360041 5115 log.cc:1196] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/bbab4368e73b4944b9853c456a4db7bd
I20240503 02:12:35.361788 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 8debae3d66514458b2f2ac04982975b8 (DEFAULT_TABLE table=default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32 [id=dfdaada115ed4a4daac3fcfe1a9bb0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.362223 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8debae3d66514458b2f2ac04982975b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.367733 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.303695 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290a61c21fa456eb3c4f9b0dbebc864 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.368714 5103 raft_consensus.cc:374] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.368818 5103 raft_consensus.cc:729] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.307986 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.320344 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b (DEFAULT_TABLE table=default.Table_5ad66186bf0948d996f2098e51b3a661 [id=43ff2e875e4e480a871ca527675be18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.350843 4519 raft_consensus.cc:3037] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.352965 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.355254 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d919b9262ac48cbb8b3f893918b5590" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.369259 4522 raft_consensus.cc:3037] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.356664 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.372687 6041 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.372838 6041 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.373725 6041 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:35.374799 4608 leader_election.cc:341] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.375603 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.377312 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 8debae3d66514458b2f2ac04982975b8 (DEFAULT_TABLE table=default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32 [id=dfdaada115ed4a4daac3fcfe1a9bb0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.377557 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.360905 4529 raft_consensus.cc:2454] T d19ed19da5b74583bfc3b21d88e937c9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.381686 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 8debae3d66514458b2f2ac04982975b8 (DEFAULT_TABLE table=default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32 [id=dfdaada115ed4a4daac3fcfe1a9bb0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.385336 6057 raft_consensus.cc:482] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.385529 6057 raft_consensus.cc:504] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.386413 6057 leader_election.cc:288] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.389982 5103 consensus_queue.cc:260] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.392434 5115 ts_tablet_manager.cc:1909] T bbab4368e73b4944b9853c456a4db7bd P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:35.393558 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 192460518d49417cb8f03ac6ec971eef (DEFAULT_TABLE table=default.Table_53cb05a18acb413295d4cfe37fb3b76d [id=739fc0c312224e5a8c60ee6d817b1879]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.394016 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192460518d49417cb8f03ac6ec971eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.399626 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 192460518d49417cb8f03ac6ec971eef (DEFAULT_TABLE table=default.Table_53cb05a18acb413295d4cfe37fb3b76d [id=739fc0c312224e5a8c60ee6d817b1879]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.401343 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 192460518d49417cb8f03ac6ec971eef (DEFAULT_TABLE table=default.Table_53cb05a18acb413295d4cfe37fb3b76d [id=739fc0c312224e5a8c60ee6d817b1879]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.401685 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 1df47adfa50a41229b5672b7c3d1794b (DEFAULT_TABLE table=default.Table_278fc530c46a46159b5cff1d43aa3a5b [id=1d74efda69c34787b5a8d227ec92cc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.402160 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df47adfa50a41229b5672b7c3d1794b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.406360 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 1df47adfa50a41229b5672b7c3d1794b (DEFAULT_TABLE table=default.Table_278fc530c46a46159b5cff1d43aa3a5b [id=1d74efda69c34787b5a8d227ec92cc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.406657 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 1df47adfa50a41229b5672b7c3d1794b (DEFAULT_TABLE table=default.Table_278fc530c46a46159b5cff1d43aa3a5b [id=1d74efda69c34787b5a8d227ec92cc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.408282 5106 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.408211 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.408505 4675 raft_consensus.cc:2454] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:35.408546 5106 ts_tablet_manager.cc:1367] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.304s user 0.032s sys 0.072s
I20240503 02:12:35.411765 5106 raft_consensus.cc:348] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.411993 5106 raft_consensus.cc:374] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.412047 5106 raft_consensus.cc:729] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.412245 5106 consensus_queue.cc:260] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.412842 5106 ts_tablet_manager.cc:1398] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:35.413219 5106 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.413617 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:35.414291 4746 leader_election.cc:341] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.414469 4746 leader_election.cc:302] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.414732 6078 raft_consensus.cc:2726] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.415374 5103 ts_tablet_manager.cc:1398] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.130s user 0.003s sys 0.000s
I20240503 02:12:35.415771 5106 tablet_bootstrap.cc:654] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:35.417681 4746 leader_election.cc:341] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.417922 4746 leader_election.cc:302] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.418325 6078 raft_consensus.cc:2726] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.419975 6107 raft_consensus.cc:504] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.420624 6107 leader_election.cc:288] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.420662 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8debae3d66514458b2f2ac04982975b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.421053 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df47adfa50a41229b5672b7c3d1794b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:35.421522 6024 consensus_meta.cc:287] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.507s user 0.003s sys 0.000s
I20240503 02:12:35.421612 6024 raft_consensus.cc:504] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.422130 6024 leader_election.cc:288] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.423291 6001 raft_consensus.cc:2726] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.423503 4807 raft_consensus.cc:2454] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:35.424005 4522 raft_consensus.cc:2454] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.424405 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192460518d49417cb8f03ac6ec971eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.433920 5114 ts_tablet_manager.cc:1888] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:35.434077 5114 log.cc:1196] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b080fc5ffc564420a0b7f2b7460ebbbb
I20240503 02:12:35.434497 5114 ts_tablet_manager.cc:1909] T b080fc5ffc564420a0b7f2b7460ebbbb P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:35.434863 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.435472 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192460518d49417cb8f03ac6ec971eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.435920 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df47adfa50a41229b5672b7c3d1794b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.436386 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8debae3d66514458b2f2ac04982975b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.440613 5106 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.440749 5106 ts_tablet_manager.cc:1367] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.005s
I20240503 02:12:35.441397 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.441612 4522 raft_consensus.cc:3037] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.441610 5106 raft_consensus.cc:348] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.441803 5106 raft_consensus.cc:374] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.441850 5106 raft_consensus.cc:729] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.442050 5106 consensus_queue.cc:260] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.444475 6041 raft_consensus.cc:482] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.444504 6001 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:35.444631 6041 raft_consensus.cc:504] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.444624 6001 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.445272 6001 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.445684 4608 leader_election.cc:302] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.445900 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:35.445977 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022b5df4a914e439fff918f70c34157" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.446100 4807 raft_consensus.cc:3037] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.446144 4805 raft_consensus.cc:2379] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 2: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:35.446283 6001 raft_consensus.cc:2781] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.446400 6001 raft_consensus.cc:686] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:35.446651 6001 consensus_queue.cc:237] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.446919 6041 leader_election.cc:288] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:35.447625 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.447839 4668 raft_consensus.cc:2379] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:35.456884 5106 ts_tablet_manager.cc:1398] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:12:35.457345 5106 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.459314 5106 tablet_bootstrap.cc:654] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.466183 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.466552 4525 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 2.
I20240503 02:12:35.467049 4608 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:35.467097 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:35.467412 6024 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:35.467433 4808 raft_consensus.cc:3037] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.467506 6024 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:35.467589 6024 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:35.467973 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ced4dfb2cf541af98d50d22d1a0f0ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.468273 4529 raft_consensus.cc:3037] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.469290 5103 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:35.471511 5103 tablet_bootstrap.cc:654] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.473793 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 56aa7303f7784d2289a8da9653d1e317 (DEFAULT_TABLE table=default.Table_61c9ba66e720449da2d10423b0c029bb [id=36d53aee1b6e4aab8a70b91caf8485de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.474249 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56aa7303f7784d2289a8da9653d1e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.474753 5106 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.474871 5106 ts_tablet_manager.cc:1367] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.015s
I20240503 02:12:35.475566 5106 raft_consensus.cc:348] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.475772 5106 raft_consensus.cc:374] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.475822 5106 raft_consensus.cc:729] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.476009 5106 consensus_queue.cc:260] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.476672 5106 ts_tablet_manager.cc:1398] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:35.476964 5106 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.479233 5106 tablet_bootstrap.cc:654] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.481709 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 56aa7303f7784d2289a8da9653d1e317 (DEFAULT_TABLE table=default.Table_61c9ba66e720449da2d10423b0c029bb [id=36d53aee1b6e4aab8a70b91caf8485de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.482546 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b080fc5ffc564420a0b7f2b7460ebbbb (table default.Table_fd0e6dfdbbc14c7599538eb05a30083c [id=e08a62f9895e4721b5524313acf78175]) successfully deleted
I20240503 02:12:35.482570 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 (DEFAULT_TABLE table=default.Table_95aad9f7fca14d9fb0d98d25071f9f08 [id=bd36072625b7410ba943f6a53d7f89a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.482926 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86a6bb791449b5a4c490663d87afd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:35.482828 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c0b9003f248b47a9952e6f995887ae0c because the tablet was not found. No further retry: Not found: Tablet not found: c0b9003f248b47a9952e6f995887ae0c
I20240503 02:12:35.483242 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 9327a9e5472b4eda854320c3a95f8e56 (table default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]) successfully deleted
I20240503 02:12:35.485220 4651 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c0539bc1c4140b09ac51ad670c0e5 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.485558 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c0539bc1c4140b09ac51ad670c0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.486722 4267 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a] alter complete (version 1)
I20240503 02:12:35.487273 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8fa27e930c344572afb5f6ba668633df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.487425 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 69756.6us
W20240503 02:12:35.487664 4746 leader_election.cc:341] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.487835 4746 leader_election.cc:302] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:35.487864 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fa27e930c344572afb5f6ba668633df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:35.488099 4746 leader_election.cc:341] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.488282 4746 leader_election.cc:302] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:35.489018 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:35.489187 4746 leader_election.cc:341] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.489216 4805 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 2.
I20240503 02:12:35.489317 4746 leader_election.cc:302] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.489729 4746 leader_election.cc:302] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:35.489830 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001fe4bb80c24d8e97d971a7130b2296" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:35.490206 5828 raft_consensus.cc:2781] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.490334 5828 raft_consensus.cc:686] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:35.490618 5828 consensus_queue.cc:237] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.491729 6079 raft_consensus.cc:2781] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.492240 5990 raft_consensus.cc:2726] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.493427 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 90128cc31d5a49fb8318274b7ced27f2 (DEFAULT_TABLE table=default.Table_3a2b0f0f386845c18712a0ddb23d9cd0 [id=dd4b84daa6e34e538ef1b6e19c30bb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.493849 4638 tablet_service.cc:1467] Processing CreateTablet for tablet e41addef3c8d4c379ca3b1b948fcd298 (DEFAULT_TABLE table=default.Table_f235f17c19bd4117b7a88a40edb5e3ef [id=a7a68c5e00f546e4a08583cd8f72210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.494024 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90128cc31d5a49fb8318274b7ced27f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.494204 4645 tablet_service.cc:1467] Processing CreateTablet for tablet efd8e0353b5f4ce484ac81d646b5323d (DEFAULT_TABLE table=default.Table_4a15903f49d74e708bd42ab681cb77a5 [id=deab2f4da9234b52b3ada0cc196afeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.494549 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 80b667e4f53c4738a974664eff59469a (DEFAULT_TABLE table=default.Table_85851cb314a247e4a318be19d0c3b97e [id=d7b72470b9bd463b95c034d01a5539a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.494822 5990 raft_consensus.cc:2781] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.494891 5990 raft_consensus.cc:482] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.494971 5990 raft_consensus.cc:3037] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.495183 4462 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:35.496269 5868 raft_consensus.cc:2781] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:12:35.496369 5868 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.496582 4462 leader_election.cc:302] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:35.501070 6013 raft_consensus.cc:3037] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.501251 5105 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.501402 5105 ts_tablet_manager.cc:1367] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.207s user 0.026s sys 0.043s
W20240503 02:12:35.501765 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b94c407d27984998a550bef0d6f48c33 has no consensus state
I20240503 02:12:35.502224 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2288fa2ae0674a20b78e456480dd514b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.502228 5105 raft_consensus.cc:348] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:35.502557 5105 raft_consensus.cc:374] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.502640 5105 raft_consensus.cc:729] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:35.502802 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2288fa2ae0674a20b78e456480dd514b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.502862 5105 consensus_queue.cc:260] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:35.503043 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f8b6599206549dd8ddf449a52c61fce consensus state has no leader
I20240503 02:12:35.503417 5105 ts_tablet_manager.cc:1398] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.503350 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_02a490be3c6544d5a09aa2551909cee3" } new_table_name: "default.Table_9091ef41bcba4103811266c2b3782904" modify_external_catalogs: true
I20240503 02:12:35.503700 5105 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.504343 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41addef3c8d4c379ca3b1b948fcd298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.504652 5937 fs_manager.cc:709] Time spent opening directory manager: real 0.372s user 0.039s sys 0.049s
I20240503 02:12:35.505275 5105 tablet_bootstrap.cc:654] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.506951 5103 tablet_bootstrap.cc:492] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:35.507084 5103 ts_tablet_manager.cc:1367] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240503 02:12:35.502799 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_885ac00da36e484f81d9bdba48c6e155" } modify_external_catalogs: true
I20240503 02:12:35.507539 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_885ac00da36e484f81d9bdba48c6e155" } modify_external_catalogs: true
I20240503 02:12:35.507910 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd8e0353b5f4ce484ac81d646b5323d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.507861 5103 raft_consensus.cc:348] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.508049 5103 raft_consensus.cc:374] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.508098 5103 raft_consensus.cc:729] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.508443 5103 consensus_queue.cc:260] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.508977 5103 ts_tablet_manager.cc:1398] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.509174 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b667e4f53c4738a974664eff59469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.511978 5103 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:35.513923 6001 raft_consensus.cc:482] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:35.514067 6001 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.514277 5159 ts_tablet_manager.cc:1888] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:35.514379 5159 log.cc:1196] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/617ad14927c1441b800a204aefc2e619
I20240503 02:12:35.514712 5159 ts_tablet_manager.cc:1909] T 617ad14927c1441b800a204aefc2e619 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:35.514678 6001 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.515143 5106 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.515261 5106 ts_tablet_manager.cc:1367] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.012s
I20240503 02:12:35.516260 5106 raft_consensus.cc:348] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.516451 5106 raft_consensus.cc:374] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.516510 5106 raft_consensus.cc:729] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.516718 5106 consensus_queue.cc:260] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.517382 5106 ts_tablet_manager.cc:1398] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:35.517688 5106 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.518540 4672 raft_consensus.cc:2454] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
W20240503 02:12:35.518922 4639 consensus_meta.cc:287] T 91e785eaaf534b8c81aa6132ab929eea P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.829s user 0.001s sys 0.002s
I20240503 02:12:35.519275 5106 tablet_bootstrap.cc:654] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.519450 4810 raft_consensus.cc:2454] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:35.522576 5103 tablet_bootstrap.cc:654] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.523313 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56aa7303f7784d2289a8da9653d1e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.524410 5105 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:35.524539 5105 ts_tablet_manager.cc:1367] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 02:12:35.525353 5105 raft_consensus.cc:348] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.525543 5105 raft_consensus.cc:374] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.525591 5105 raft_consensus.cc:729] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.525828 5105 consensus_queue.cc:260] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.526345 5105 ts_tablet_manager.cc:1398] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:35.526615 5105 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:35.527611 5103 tablet_bootstrap.cc:492] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:35.527738 5103 ts_tablet_manager.cc:1367] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240503 02:12:35.527801 5106 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.527925 5106 ts_tablet_manager.cc:1367] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 02:12:35.528215 5105 tablet_bootstrap.cc:654] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.528468 5103 raft_consensus.cc:348] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.528662 5103 raft_consensus.cc:374] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.528718 5103 raft_consensus.cc:729] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.528714 5106 raft_consensus.cc:348] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.528870 5106 raft_consensus.cc:374] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.528971 5106 raft_consensus.cc:729] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.529189 5106 consensus_queue.cc:260] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.528903 5103 consensus_queue.cc:260] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.529906 5106 ts_tablet_manager.cc:1398] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.530236 5106 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.531886 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1f7ba66d347245859fec5ef1be88f98f" } new_table_name: "default.Table_b353c84797f245b08daaeffe1d2615fb" modify_external_catalogs: true
I20240503 02:12:35.532511 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8c6c80efe35343989f46a2c181f38129
I20240503 02:12:35.532680 5106 tablet_bootstrap.cc:654] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.533612 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9801d79108b449c69664c004a6c352a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.531911 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e7dad4cf88048efa489e61c2595ad65" } modify_external_catalogs: true
W20240503 02:12:35.534396 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9801d79108b449c69664c004a6c352a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.534483 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3e7dad4cf88048efa489e61c2595ad65" } modify_external_catalogs: true
I20240503 02:12:35.501787 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8f8f3c9fd776473b8dd8eed18d12c897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.537436 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f8f3c9fd776473b8dd8eed18d12c897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.540232 6168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:35.541591 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6c80efe35343989f46a2c181f38129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.542901 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 540) took 4568 ms. Trace:
I20240503 02:12:35.543020 5114 rpcz_store.cc:276] 0503 02:12:30.974362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.070127 (+1095765us) service_pool.cc:225] Handling call
0503 02:12:35.282900 (+3212773us) ts_tablet_manager.cc:1120] Deleting tablet b080fc5ffc564420a0b7f2b7460ebbbb
0503 02:12:35.282910 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.324834 (+ 41924us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.433759 (+108925us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.542864 (+109105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212482}
I20240503 02:12:35.543581 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 541) took 4569 ms. Trace:
I20240503 02:12:35.543668 5114 rpcz_store.cc:276] 0503 02:12:30.974464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.070446 (+1095982us) service_pool.cc:225] Handling call
0503 02:12:35.543417 (+3472971us) ts_tablet_manager.cc:1120] Deleting tablet 5b91a3f3cad74f6aa6035a013cca651f
0503 02:12:35.543432 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.543563 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3472789}
I20240503 02:12:35.543987 5114 tablet_replica.cc:330] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:35.544217 5114 raft_consensus.cc:2229] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:35.544440 5114 raft_consensus.cc:2258] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.553009 5937 fs_manager.cc:725] Time spent opening block manager: real 0.046s user 0.003s sys 0.000s
I20240503 02:12:35.553257 5937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
uuid: "145afaa7548f4b7fafb23f20276f0cb4"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-ddxx"
I20240503 02:12:35.555583 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.556015 4608 leader_election.cc:302] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:35.556205 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.556469 6075 raft_consensus.cc:2781] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.556592 6075 raft_consensus.cc:686] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:35.556835 6075 consensus_queue.cc:237] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:35.557570 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7efef931ca554ae18579bbea8d1c04f7 consensus state has no leader
I20240503 02:12:35.558017 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_75ace3349fd14ae992051f80c21e3896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.558589 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75ace3349fd14ae992051f80c21e3896 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.559906 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2
I20240503 02:12:35.560657 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5df26c990f414242ac526c32679385fe" } modify_external_catalogs: true
I20240503 02:12:35.560832 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5df26c990f414242ac526c32679385fe" } modify_external_catalogs: true
I20240503 02:12:35.561872 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6623c5960394449394b0bc3dae42bdd6" } new_table_name: "default.Table_04dad7f94d5f493da9ac2818393cf141" modify_external_catalogs: true
I20240503 02:12:35.554926 5937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:35.566000 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:35.570355 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 001fe4bb80c24d8e97d971a7130b2296 consensus state has no leader
I20240503 02:12:35.570797 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3e94a9c61cb74cc08ca5867ae85ecdf0
I20240503 02:12:35.571955 4522 raft_consensus.cc:2454] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:35.572649 4608 leader_election.cc:302] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:35.573163 6107 raft_consensus.cc:2781] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.573324 6107 raft_consensus.cc:686] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:35.573634 6107 consensus_queue.cc:237] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.575685 6107 raft_consensus.cc:482] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.575812 6107 raft_consensus.cc:504] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.576407 6107 leader_election.cc:288] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.579092 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.586115 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 65774b69a8bd4664a6b55b84eba37f7f
W20240503 02:12:35.587901 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 984fe4640980407aa03c16c14a1303c1 consensus state has no leader
I20240503 02:12:35.589012 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:35.589915 6057 consensus_queue.cc:1035] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.590714 6057 raft_consensus.cc:482] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.590873 6057 raft_consensus.cc:504] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.591667 6057 leader_election.cc:288] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.593115 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:35.593884 4608 leader_election.cc:341] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.595999 5106 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.596184 5106 ts_tablet_manager.cc:1367] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20240503 02:12:35.597033 5106 raft_consensus.cc:348] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.597318 5106 raft_consensus.cc:374] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.597388 5106 raft_consensus.cc:729] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.597643 5106 consensus_queue.cc:260] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.598327 5106 ts_tablet_manager.cc:1398] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:35.598664 5106 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:35.602332 5114 ts_tablet_manager.cc:1875] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.602882 5106 tablet_bootstrap.cc:654] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.609396 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 (DEFAULT_TABLE table=default.Table_95aad9f7fca14d9fb0d98d25071f9f08 [id=bd36072625b7410ba943f6a53d7f89a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.610419 4509 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c0539bc1c4140b09ac51ad670c0e5 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.611295 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 90128cc31d5a49fb8318274b7ced27f2 (DEFAULT_TABLE table=default.Table_3a2b0f0f386845c18712a0ddb23d9cd0 [id=dd4b84daa6e34e538ef1b6e19c30bb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.619329 5999 raft_consensus.cc:504] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.620492 5999 leader_election.cc:288] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:35.621260 4811 consensus_meta.cc:287] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.565s user 0.003s sys 0.001s
I20240503 02:12:35.621371 4811 raft_consensus.cc:2454] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:35.626482 4462 leader_election.cc:302] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:35.626981 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.627262 4668 raft_consensus.cc:2454] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:35.633694 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b080fc5ffc564420a0b7f2b7460ebbbb (table default.Table_fd0e6dfdbbc14c7599538eb05a30083c [id=e08a62f9895e4721b5524313acf78175]) successfully deleted
W20240503 02:12:35.634161 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5b91a3f3cad74f6aa6035a013cca651f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b91a3f3cad74f6aa6035a013cca651f already in progress: creating tablet
I20240503 02:12:35.634020 6057 consensus_queue.cc:1035] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.634578 4746 leader_election.cc:302] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:35.635553 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.635485 6057 raft_consensus.cc:2781] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.636241 6057 raft_consensus.cc:482] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.636368 6057 raft_consensus.cc:3037] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:35.647658 4746 leader_election.cc:341] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.651386 6183 raft_consensus.cc:2781] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.657506 6183 raft_consensus.cc:686] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:35.659760 4501 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6c80efe35343989f46a2c181f38129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.659850 4503 tablet_service.cc:1467] Processing CreateTablet for tablet efd8e0353b5f4ce484ac81d646b5323d (DEFAULT_TABLE table=default.Table_4a15903f49d74e708bd42ab681cb77a5 [id=deab2f4da9234b52b3ada0cc196afeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.660174 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.661520 6183 consensus_queue.cc:237] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.682811 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ea4dac17dca3442d8934ac357779b78a
I20240503 02:12:35.687192 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4dac17dca3442d8934ac357779b78a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:35.698407 4646 consensus_meta.cc:287] T 0a8c57ddf3d8428e8fa5ffb061dc054b P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.520s user 0.004s sys 0.000s
I20240503 02:12:35.704061 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.704231 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_914edf6b009344fba3c70a9fde8b6aee" } new_table_name: "default.Table_dbaf26d9e0e4472b9a762b7c9e0f0361" modify_external_catalogs: true
I20240503 02:12:35.706010 6182 consensus_queue.cc:1035] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.706373 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_41d45c372f894144a4e54926d63a14b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.707310 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d45c372f894144a4e54926d63a14b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.660336 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 80b667e4f53c4738a974664eff59469a (DEFAULT_TABLE table=default.Table_85851cb314a247e4a318be19d0c3b97e [id=d7b72470b9bd463b95c034d01a5539a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.660518 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.660825 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.710927 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a6a57c6d925744ca8d5983412f5f516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.711499 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6a57c6d925744ca8d5983412f5f516f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.710953 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_df82f492a3db4e1480369126b97c1169" } new_table_name: "default.Table_2e0422a1f16245238c15c03daefcbe32" modify_external_catalogs: true
W20240503 02:12:35.719092 4608 leader_election.cc:341] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.721823 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 552) took 2906 ms. Trace:
I20240503 02:12:35.722056 5115 rpcz_store.cc:276] 0503 02:12:32.815197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.815291 (+ 94us) service_pool.cc:225] Handling call
0503 02:12:34.875171 (+2059880us) ts_tablet_manager.cc:1120] Deleting tablet bbab4368e73b4944b9853c456a4db7bd
0503 02:12:34.875184 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.064886 (+189702us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.359752 (+294866us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.721787 (+362035us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2021561}
I20240503 02:12:35.723004 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 561) took 2688 ms. Trace:
I20240503 02:12:35.723191 5115 rpcz_store.cc:276] 0503 02:12:33.035030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.035627 (+ 597us) service_pool.cc:225] Handling call
0503 02:12:35.722809 (+2687182us) ts_tablet_manager.cc:1120] Deleting tablet 0700d4aed530413983381d1dc4f78ef9
0503 02:12:35.722824 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.722983 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2686709}
I20240503 02:12:35.723628 5115 tablet_replica.cc:330] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:35.724030 5115 raft_consensus.cc:2229] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:35.724535 5115 raft_consensus.cc:2258] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.726023 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 56aa7303f7784d2289a8da9653d1e317 (DEFAULT_TABLE table=default.Table_61c9ba66e720449da2d10423b0c029bb [id=36d53aee1b6e4aab8a70b91caf8485de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.692806 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:35.731654 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8f8b6599206549dd8ddf449a52c61fce (table default.Table_453e312025324be6a4d15ae5116a46fa [id=c11dd6cb733044ef94c59992e98601f1], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8f8b6599206549dd8ddf449a52c61fce consensus state has no leader
I20240503 02:12:35.733234 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet bbab4368e73b4944b9853c456a4db7bd (table default.Table_8925d918e3de4d60b2f13a0c0e06f5be [id=08acf0302ba442be9479d02c5c27b12a]) successfully deleted
W20240503 02:12:35.733829 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0700d4aed530413983381d1dc4f78ef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700d4aed530413983381d1dc4f78ef9 already in progress: creating tablet
I20240503 02:12:35.735458 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fb49809c3ca04cf9ac1cad7770bd1897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.735716 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 335f896171b54b67ad44cb8e161003ce consensus state has no leader
W20240503 02:12:35.736092 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb49809c3ca04cf9ac1cad7770bd1897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.738327 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9947777ee256419394d206f28a1a6e11" } modify_external_catalogs: true
I20240503 02:12:35.738605 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9947777ee256419394d206f28a1a6e11" } modify_external_catalogs: true
I20240503 02:12:35.739419 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_73be651ef93740b19d0d19abe2aff424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.739771 6075 raft_consensus.cc:482] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.739910 6075 raft_consensus.cc:504] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:35.740162 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73be651ef93740b19d0d19abe2aff424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.740506 6075 leader_election.cc:288] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.744604 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4dac17dca3442d8934ac357779b78a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:35.675276 4491 tablet_service.cc:1467] Processing CreateTablet for tablet e41addef3c8d4c379ca3b1b948fcd298 (DEFAULT_TABLE table=default.Table_f235f17c19bd4117b7a88a40edb5e3ef [id=a7a68c5e00f546e4a08583cd8f72210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.752864 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d271d2c112ba44b68d2933a6ebfe7a18" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.753232 4522 raft_consensus.cc:2379] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:35.756732 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86a6bb791449b5a4c490663d87afd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.757596 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_950e46a8a847487e902ad641c4895f8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:35.758322 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_950e46a8a847487e902ad641c4895f8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.759394 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b353c84797f245b08daaeffe1d2615fb" } new_table_name: "default.Table_b6a18c634a384042bb2ca1992f701886" modify_external_catalogs: true
I20240503 02:12:35.766875 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7d5419d4bbf34b67bcc05cf32663c2df" } modify_external_catalogs: true
I20240503 02:12:35.767112 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7d5419d4bbf34b67bcc05cf32663c2df" } modify_external_catalogs: true
I20240503 02:12:35.767783 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_71333c41521440e29744c283c842b895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:35.768107 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c0539bc1c4140b09ac51ad670c0e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:35.768362 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71333c41521440e29744c283c842b895 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:35.768884 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd8e0353b5f4ce484ac81d646b5323d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.769408 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90128cc31d5a49fb8318274b7ced27f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:35.770291 4665 consensus_meta.cc:287] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81: Time spent flushing consensus metadata: real 0.524s user 0.001s sys 0.003s
I20240503 02:12:35.770382 4665 raft_consensus.cc:2454] T d19ed19da5b74583bfc3b21d88e937c9 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.770905 4807 raft_consensus.cc:2454] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:35.774744 6013 raft_consensus.cc:2726] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.781889 6177 raft_consensus.cc:482] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.782070 6177 raft_consensus.cc:504] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.782922 6177 leader_election.cc:288] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.788439 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 555) took 3134 ms. Trace:
I20240503 02:12:35.788564 5159 rpcz_store.cc:276] 0503 02:12:32.653649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.653705 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:35.331627 (+2677922us) ts_tablet_manager.cc:1120] Deleting tablet 617ad14927c1441b800a204aefc2e619
0503 02:12:35.331635 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.420085 (+ 88450us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.514170 (+ 94085us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.788409 (+274239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2677658}
I20240503 02:12:35.789040 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 556) took 3066 ms. Trace:
I20240503 02:12:35.789119 5159 rpcz_store.cc:276] 0503 02:12:32.722465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.722668 (+ 203us) service_pool.cc:225] Handling call
0503 02:12:35.788871 (+3066203us) ts_tablet_manager.cc:1120] Deleting tablet 17a24b5c83b649e79644b08253d851d6
0503 02:12:35.788879 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.789019 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3053992}
I20240503 02:12:35.789469 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 557) took 3057 ms. Trace:
I20240503 02:12:35.789536 5159 rpcz_store.cc:276] 0503 02:12:32.732278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.732352 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:35.789349 (+3056997us) ts_tablet_manager.cc:1120] Deleting tablet 552dad16f30a410280c87a6c5ef0eaa4
0503 02:12:35.789357 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.789452 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3054429}
I20240503 02:12:35.789858 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 558) took 3005 ms. Trace:
I20240503 02:12:35.789923 5159 rpcz_store.cc:276] 0503 02:12:32.784533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.784599 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:35.789730 (+3005131us) ts_tablet_manager.cc:1120] Deleting tablet 50a43e428b804729adf9d8a3133c90ff
0503 02:12:35.789738 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.789841 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004891}
I20240503 02:12:35.790220 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 566) took 2751 ms. Trace:
I20240503 02:12:35.790282 5159 rpcz_store.cc:276] 0503 02:12:33.038795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.063829 (+ 25034us) service_pool.cc:225] Handling call
0503 02:12:35.790131 (+2726302us) ts_tablet_manager.cc:1120] Deleting tablet 4c0745a55223475893edd01d904a223b
0503 02:12:35.790141 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.790205 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2725865}
I20240503 02:12:35.790555 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 565) took 2755 ms. Trace:
I20240503 02:12:35.790639 5159 rpcz_store.cc:276] 0503 02:12:33.035609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.064321 (+ 28712us) service_pool.cc:225] Handling call
0503 02:12:35.790473 (+2726152us) ts_tablet_manager.cc:1120] Deleting tablet 0700d4aed530413983381d1dc4f78ef9
0503 02:12:35.790481 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.790543 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2725940}
I20240503 02:12:35.790884 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 567) took 2592 ms. Trace:
I20240503 02:12:35.790938 5159 rpcz_store.cc:276] 0503 02:12:33.198194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.198255 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:35.790807 (+2592552us) ts_tablet_manager.cc:1120] Deleting tablet 46005e39ab284da6bf1bb9565318a2f1
0503 02:12:35.790814 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.790874 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592318}
I20240503 02:12:35.791214 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 568) took 2576 ms. Trace:
I20240503 02:12:35.791272 5159 rpcz_store.cc:276] 0503 02:12:33.214985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.215352 (+ 367us) service_pool.cc:225] Handling call
0503 02:12:35.791135 (+2575783us) ts_tablet_manager.cc:1120] Deleting tablet 818788a24cd1419980c0d622fb05683b
0503 02:12:35.791142 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.791202 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575542}
I20240503 02:12:35.791486 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 569) took 2573 ms. Trace:
I20240503 02:12:35.791533 5159 rpcz_store.cc:276] 0503 02:12:33.218514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.218734 (+ 220us) service_pool.cc:225] Handling call
0503 02:12:35.791432 (+2572698us) ts_tablet_manager.cc:1120] Deleting tablet 9327a9e5472b4eda854320c3a95f8e56
0503 02:12:35.791438 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.791476 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2572438}
I20240503 02:12:35.791718 5159 tablet_replica.cc:330] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:35.791872 5159 raft_consensus.cc:2229] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:35.792037 5159 raft_consensus.cc:2258] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.792543 5159 ts_tablet_manager.cc:1875] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.797681 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41addef3c8d4c379ca3b1b948fcd298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.798586 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56aa7303f7784d2289a8da9653d1e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.815773 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b667e4f53c4738a974664eff59469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.837538 6057 raft_consensus.cc:504] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.838697 6057 leader_election.cc:288] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.839032 5159 ts_tablet_manager.cc:1888] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:35.839128 5159 log.cc:1196] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2f053537029a44d5a519b612d09d19a1
I20240503 02:12:35.839455 5159 ts_tablet_manager.cc:1909] T 2f053537029a44d5a519b612d09d19a1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:35.839692 4673 raft_consensus.cc:2454] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:35.849419 4635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 109) took 1560 ms. Trace:
I20240503 02:12:35.849531 4635 rpcz_store.cc:276] 0503 02:12:34.288988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.291017 (+ 2029us) service_pool.cc:225] Handling call
0503 02:12:34.515202 (+224185us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x6160001a0c30. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9aa2f628 00007f1c9d42d988 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:34.750560 (+235358us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x616000143e54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:34.816378 (+ 65818us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x616000190354. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.047717 (+231339us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x616000125254. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.134884 (+ 87167us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x6160001d0854. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.422012 (+287128us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x616000026554. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.777490 (+355478us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x616000171a54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.840829 (+ 63339us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x616000151654. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:35.849344 (+ 8515us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3504599168}
I20240503 02:12:35.854095 6041 raft_consensus.cc:482] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.854494 6041 raft_consensus.cc:504] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.856403 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.857385 6075 raft_consensus.cc:482] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.857543 6075 raft_consensus.cc:504] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.858086 6041 leader_election.cc:288] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:35.858273 6075 leader_election.cc:288] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.865155 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:35.866675 4608 leader_election.cc:341] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.867513 6075 consensus_queue.cc:1035] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.876760 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b3dcd45817264953aa80592e5bd4c4e4" } new_table_name: "default.Table_f973292ca29f4783af48772d007e24de" modify_external_catalogs: true
W20240503 02:12:35.894935 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 160685us
I20240503 02:12:35.902365 5115 ts_tablet_manager.cc:1875] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.908536 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:35.909785 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:35.924276 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 570) took 2705 ms. Trace:
I20240503 02:12:35.924422 5159 rpcz_store.cc:276] 0503 02:12:33.218675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.219626 (+ 951us) service_pool.cc:225] Handling call
0503 02:12:35.791683 (+2572057us) ts_tablet_manager.cc:1120] Deleting tablet 2f053537029a44d5a519b612d09d19a1
0503 02:12:35.791689 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.832804 (+ 41115us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.838924 (+ 6120us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:35.924247 (+ 85323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2571782}
I20240503 02:12:35.924818 5159 tablet_replica.cc:330] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:35.925089 5159 raft_consensus.cc:2229] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:35.925406 5159 raft_consensus.cc:2258] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:35.926002 5159 ts_tablet_manager.cc:1875] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:35.929195 6024 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.929823 6024 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.942499 5990 raft_consensus.cc:504] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.943205 5990 leader_election.cc:288] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.947382 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.949134 6079 raft_consensus.cc:686] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:35.950712 4529 raft_consensus.cc:2454] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:35.951695 4608 leader_election.cc:302] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:35.951975 6024 raft_consensus.cc:2781] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.952105 6024 raft_consensus.cc:686] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:35.952189 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:35.952504 4673 raft_consensus.cc:3037] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.952396 6024 consensus_queue.cc:237] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.966542 5106 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:35.966871 5106 ts_tablet_manager.cc:1367] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.368s user 0.020s sys 0.045s
I20240503 02:12:35.968092 5106 raft_consensus.cc:348] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.968851 5106 raft_consensus.cc:374] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.969003 5106 raft_consensus.cc:729] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.969375 5106 consensus_queue.cc:260] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.971411 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:35.971833 5990 raft_consensus.cc:482] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.972014 5990 raft_consensus.cc:504] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.972465 6078 raft_consensus.cc:482] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.972590 6078 raft_consensus.cc:504] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.972829 5990 leader_election.cc:288] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:35.972990 5990 raft_consensus.cc:482] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:35.973086 4746 leader_election.cc:341] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:35.973104 5990 raft_consensus.cc:504] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:35.973863 5990 leader_election.cc:288] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:35.974730 4735 kernel_stack_watchdog.cc:198] Thread 5105 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 255ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fceda955980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fced75c3d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fced75e47ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fced75e03ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fced75e54ff kudu::ThreadPool::Submit() at ??:0
@ 0x7fcee278a4ad kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fcee278949f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fcee2789070 kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fcee2790f85 kudu::log::Log::Init() at ??:0
@ 0x7fcee27902e1 kudu::log::Log::Open() at ??:0
@ 0x7fcee4087fd1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fcee408433c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fcee4082abf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20240503 02:12:35.975185 6078 leader_election.cc:288] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:35.982539 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 001fe4bb80c24d8e97d971a7130b2296 consensus state has no leader
W20240503 02:12:35.983530 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c2a8190210439fb7589cb02ec82bc2 consensus state has no leader
I20240503 02:12:36.003566 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:36.004236 4746 leader_election.cc:341] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.004351 4746 leader_election.cc:302] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:36.004714 4746 leader_election.cc:341] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.005149 5990 raft_consensus.cc:2726] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:35.949692 6079 consensus_queue.cc:237] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.012984 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ac84b01b9b94262b9e66dcfb42ded1e" } modify_external_catalogs: true
I20240503 02:12:36.013235 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8ac84b01b9b94262b9e66dcfb42ded1e" } modify_external_catalogs: true
I20240503 02:12:36.014389 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:36.014663 4811 raft_consensus.cc:2379] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 2: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:36.014678 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d5f2e666523c426c8f54c9f677fbdd3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:36.015308 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5f2e666523c426c8f54c9f677fbdd3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.015780 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:36.015734 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9714c86da62e48489868fed877679b05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:36.015965 4811 raft_consensus.cc:2454] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:36.016259 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9714c86da62e48489868fed877679b05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.016366 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:36.016525 4811 raft_consensus.cc:3037] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.017138 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5f22a82442be4541886bad9e8ada3df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:36.017670 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f22a82442be4541886bad9e8ada3df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.017726 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5ad66186bf0948d996f2098e51b3a661" } modify_external_catalogs: true
I20240503 02:12:36.017913 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5ad66186bf0948d996f2098e51b3a661" } modify_external_catalogs: true
I20240503 02:12:36.023386 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:36.003818 4807 raft_consensus.cc:2454] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:36.024111 4267 catalog_manager.cc:5484] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:36.025401 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:36.031510 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:36.036404 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 56498.1us
I20240503 02:12:36.038048 4504 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bba49712445db6d784b6ed94dd7e (DEFAULT_TABLE table=default.Table_94e778467d034214b379477ebfe9694e [id=e42001ef694841f780796904905e3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.034595 6174 raft_consensus.cc:482] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.041522 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b21a3010a4470288fbdd684ef64e77" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.043062 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e9faaa6a584ec38b9da9d2670e47f3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.044400 6041 consensus_queue.cc:1035] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.045464 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cdd9fdae843fe86c779e8eb67bfe2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.047652 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9359d8faab497c86d958f7ec9506de (DEFAULT_TABLE table=default.Table_366708ec2da84f23933deff6182b9fe5 [id=0d8b346069de44f5bc7dfbd98ee5a7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.049558 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 92126108432e4450ba30cfde8ca8618f (DEFAULT_TABLE table=default.Table_a9135679f7fb423cb1d8d69c3430feb0 [id=e1500edee3984ef88898d6fac0f50de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.052701 6041 raft_consensus.cc:482] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.052804 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 (DEFAULT_TABLE table=default.Table_95aad9f7fca14d9fb0d98d25071f9f08 [id=bd36072625b7410ba943f6a53d7f89a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.052846 6041 raft_consensus.cc:504] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.054189 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 90128cc31d5a49fb8318274b7ced27f2 (DEFAULT_TABLE table=default.Table_3a2b0f0f386845c18712a0ddb23d9cd0 [id=dd4b84daa6e34e538ef1b6e19c30bb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.055727 6041 leader_election.cc:288] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.057713 4774 tablet_service.cc:1467] Processing CreateTablet for tablet efd8e0353b5f4ce484ac81d646b5323d (DEFAULT_TABLE table=default.Table_4a15903f49d74e708bd42ab681cb77a5 [id=deab2f4da9234b52b3ada0cc196afeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.058283 4462 leader_election.cc:302] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:36.058779 6041 raft_consensus.cc:2781] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:36.058867 6041 raft_consensus.cc:482] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f50719d94564844b15e1a9211d48a81)
I20240503 02:12:36.058950 6041 raft_consensus.cc:3037] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:36.059376 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:36.059646 4807 raft_consensus.cc:2454] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:36.059877 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.060034 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6c80efe35343989f46a2c181f38129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.059651 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:35.994717 5999 consensus_queue.cc:1035] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.058125 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:36.060427 4806 raft_consensus.cc:2379] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 2: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:36.017693 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:36.060843 4665 raft_consensus.cc:3037] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.019065 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cdd9fdae843fe86c779e8eb67bfe2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.061210 4668 raft_consensus.cc:2454] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:36.037606 4652 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bba49712445db6d784b6ed94dd7e (DEFAULT_TABLE table=default.Table_94e778467d034214b379477ebfe9694e [id=e42001ef694841f780796904905e3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.040896 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b21a3010a4470288fbdd684ef64e77" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.042470 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e9faaa6a584ec38b9da9d2670e47f3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.062139 4675 raft_consensus.cc:2454] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:36.038995 6174 raft_consensus.cc:504] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:36.046818 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9359d8faab497c86d958f7ec9506de (DEFAULT_TABLE table=default.Table_366708ec2da84f23933deff6182b9fe5 [id=0d8b346069de44f5bc7dfbd98ee5a7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.048708 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 92126108432e4450ba30cfde8ca8618f (DEFAULT_TABLE table=default.Table_a9135679f7fb423cb1d8d69c3430feb0 [id=e1500edee3984ef88898d6fac0f50de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.054885 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.062909 4670 raft_consensus.cc:2379] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
W20240503 02:12:36.063529 4747 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.002s sys 0.007s
I20240503 02:12:36.065012 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.065596 4786 tablet_service.cc:1467] Processing CreateTablet for tablet e41addef3c8d4c379ca3b1b948fcd298 (DEFAULT_TABLE table=default.Table_f235f17c19bd4117b7a88a40edb5e3ef [id=a7a68c5e00f546e4a08583cd8f72210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.066474 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 80b667e4f53c4738a974664eff59469a (DEFAULT_TABLE table=default.Table_85851cb314a247e4a318be19d0c3b97e [id=d7b72470b9bd463b95c034d01a5539a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.066890 4784 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c0539bc1c4140b09ac51ad670c0e5 (DEFAULT_TABLE table=default.Table_a8465a99526148a0b0b88f92c5d1e3a0 [id=49c604a2f5354ff7b6fecd973dc2d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.066955 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4dac17dca3442d8934ac357779b78a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.071091 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 52f167be4c1c43909eacaf5421882f7f (DEFAULT_TABLE table=default.Table_d64beb8ee02743139edc02931b1f97ee [id=c164713a85064ac0b6099b40f3a32473]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.072342 4655 tablet_service.cc:1467] Processing CreateTablet for tablet cd915eae241448f2abe14b04640174b0 (DEFAULT_TABLE table=default.Table_44989b3752354e3e96f57a2f15a27ec6 [id=ddf91007d86046d4b6d6db7a47bb8e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.073565 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 0db84f8ed09342ea8b1c3b00b2036c50 (DEFAULT_TABLE table=default.Table_acdeb2544a5243f184185b6c97b607f8 [id=c88bf7514ead43e68bfe83e7cdbfe730]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.073634 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 151f925dcb304594b60c85d850b83533 (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.074468 4636 tablet_service.cc:1467] Processing CreateTablet for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b (DEFAULT_TABLE table=default.Table_91951087060348328675a5350c9a1dea [id=4706d758b1094d6081bf011c9415c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.075454 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 5e534255f170416690c18ca6bd571117 (DEFAULT_TABLE table=default.Table_b1103caea8f04d32ba60345676c5fa63 [id=d11bb162d01f44f5afab631c1ede7562]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.076375 4646 tablet_service.cc:1467] Processing CreateTablet for tablet d3b9341be62a4275ad6e4c7d7b745621 (DEFAULT_TABLE table=default.Table_1c52b1bbf50148c5b4753da4f8fe72fc [id=c65010247bfd4c7ab61e95464f125c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.077301 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8656a886564f06931039b516162b51 (DEFAULT_TABLE table=default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b [id=f4b10783625e48ad95d347379e85460f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.078594 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db84f8ed09342ea8b1c3b00b2036c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.078799 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151f925dcb304594b60c85d850b83533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.079402 5159 ts_tablet_manager.cc:1888] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.084400 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b9341be62a4275ad6e4c7d7b745621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.085438 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bba49712445db6d784b6ed94dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.085796 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd915eae241448f2abe14b04640174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.086367 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92126108432e4450ba30cfde8ca8618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.087934 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f167be4c1c43909eacaf5421882f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.088768 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9359d8faab497c86d958f7ec9506de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.089277 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.089735 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e534255f170416690c18ca6bd571117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.093180 4673 raft_consensus.cc:2454] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:36.097108 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8656a886564f06931039b516162b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.129329 4519 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 2.
I20240503 02:12:36.130429 4608 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:36.130762 5999 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:36.130894 5999 raft_consensus.cc:686] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:36.130862 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 52f167be4c1c43909eacaf5421882f7f (DEFAULT_TABLE table=default.Table_d64beb8ee02743139edc02931b1f97ee [id=c164713a85064ac0b6099b40f3a32473]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.131165 5999 consensus_queue.cc:237] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.151458 4505 tablet_service.cc:1467] Processing CreateTablet for tablet cd915eae241448f2abe14b04640174b0 (DEFAULT_TABLE table=default.Table_44989b3752354e3e96f57a2f15a27ec6 [id=ddf91007d86046d4b6d6db7a47bb8e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.152279 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 0db84f8ed09342ea8b1c3b00b2036c50 (DEFAULT_TABLE table=default.Table_acdeb2544a5243f184185b6c97b607f8 [id=c88bf7514ead43e68bfe83e7cdbfe730]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.152987 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 151f925dcb304594b60c85d850b83533 (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.153695 4502 tablet_service.cc:1467] Processing CreateTablet for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b (DEFAULT_TABLE table=default.Table_91951087060348328675a5350c9a1dea [id=4706d758b1094d6081bf011c9415c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.163079 4499 tablet_service.cc:1467] Processing CreateTablet for tablet d3b9341be62a4275ad6e4c7d7b745621 (DEFAULT_TABLE table=default.Table_1c52b1bbf50148c5b4753da4f8fe72fc [id=c65010247bfd4c7ab61e95464f125c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.165787 5114 ts_tablet_manager.cc:1888] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.165907 5114 log.cc:1196] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/19fb0929bd234788b5313d96d2e4845c
I20240503 02:12:36.169821 4665 raft_consensus.cc:2454] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:36.171036 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f167be4c1c43909eacaf5421882f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.171285 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.171797 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151f925dcb304594b60c85d850b83533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.170943 5937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:36.172286 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b9341be62a4275ad6e4c7d7b745621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.172662 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db84f8ed09342ea8b1c3b00b2036c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.173024 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd915eae241448f2abe14b04640174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.173393 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92126108432e4450ba30cfde8ca8618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.174724 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bba49712445db6d784b6ed94dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:36.183797 4447 kernel_stack_watchdog.cc:198] Thread 5103 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 336ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f13966d2980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f1393340d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f13933617ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f139335d3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f139335b84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f139eb554f0 kudu::consensus::RaftConsensus::MarkDirty() at ??:0
@ 0x7f139eb4e1b0 kudu::consensus::RaftConsensus::Start() at ??:0
@ 0x7f139fe639ce kudu::tablet::TabletReplica::Start() at ??:0
@ 0x7f13a28f5554 kudu::tserver::TSTabletManager::OpenTablet() at ??:0
@ 0x7f13a290b49b kudu::tserver::TSTabletManager::CreateNewTablet()::$_11::operator()() at ??:0
@ 0x7f13a290b1de std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1393363b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 02:12:36.184712 5937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:36.189767 5937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:36.195683 5999 raft_consensus.cc:482] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.195878 5999 raft_consensus.cc:504] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.196700 5999 leader_election.cc:288] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.197589 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:36.200574 4608 leader_election.cc:341] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.208029 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9359d8faab497c86d958f7ec9506de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.215391 6041 raft_consensus.cc:504] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.227571 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41dce88919456aa846dbdcf5e2759f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:36.237360 6041 leader_election.cc:288] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.245319 5999 raft_consensus.cc:482] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.245518 5999 raft_consensus.cc:504] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.246501 5999 leader_election.cc:288] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.247498 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:36.248569 4608 leader_election.cc:341] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:36.256295 4608 consensus_peers.cc:485] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:36.297171 5114 ts_tablet_manager.cc:1909] T 19fb0929bd234788b5313d96d2e4845c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.298182 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 542) took 4762 ms. Trace:
I20240503 02:12:36.300860 5106 ts_tablet_manager.cc:1398] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.334s user 0.003s sys 0.000s
I20240503 02:12:36.301465 5106 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:36.305194 4746 leader_election.cc:341] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.314193 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.314380 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.323057 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.323973 6174 leader_election.cc:288] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:36.326651 4747 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.003s sys 0.007s
I20240503 02:12:36.328200 5828 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.328712 6078 raft_consensus.cc:482] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.328765 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b161628065704c798020edbe098844c2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:36.298285 5114 rpcz_store.cc:276] 0503 02:12:31.535493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.070633 (+535140us) service_pool.cc:225] Handling call
0503 02:12:35.543935 (+3473302us) ts_tablet_manager.cc:1120] Deleting tablet 19fb0929bd234788b5313d96d2e4845c
0503 02:12:35.543945 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.823491 (+279546us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.165660 (+342169us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.298149 (+132489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3473133}
I20240503 02:12:36.329785 5159 log.cc:1196] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:36.330085 5114 tablet_replica.cc:330] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:36.330250 5159 ts_tablet_manager.cc:1909] T 36a14c4f785f4514ae74b8a0b3073503 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:36.330627 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90128cc31d5a49fb8318274b7ced27f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.330600 5114 raft_consensus.cc:2229] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.331194 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 574) took 3110 ms. Trace:
I20240503 02:12:36.331223 5114 raft_consensus.cc:2258] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.331322 5159 rpcz_store.cc:276] 0503 02:12:33.219913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.219962 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:35.924757 (+2704795us) ts_tablet_manager.cc:1120] Deleting tablet 36a14c4f785f4514ae74b8a0b3073503
0503 02:12:35.924768 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:35.932980 (+ 8212us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.079288 (+146308us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.330577 (+251289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704645}
I20240503 02:12:36.328881 6078 raft_consensus.cc:504] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.332834 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c0539bc1c4140b09ac51ad670c0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.333181 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b667e4f53c4738a974664eff59469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.332860 5159 tablet_replica.cc:330] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:36.337550 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd8e0353b5f4ce484ac81d646b5323d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.338642 6078 leader_election.cc:288] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.339953 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41addef3c8d4c379ca3b1b948fcd298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.340955 5159 raft_consensus.cc:2229] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.341966 5159 raft_consensus.cc:2258] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.343225 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 5e534255f170416690c18ca6bd571117 (DEFAULT_TABLE table=default.Table_b1103caea8f04d32ba60345676c5fa63 [id=d11bb162d01f44f5afab631c1ede7562]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.343830 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e534255f170416690c18ca6bd571117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.349418 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 19fb0929bd234788b5313d96d2e4845c (table default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]) successfully deleted
I20240503 02:12:36.351168 5159 ts_tablet_manager.cc:1875] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.351660 4491 tablet_service.cc:1467] Processing CreateTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 (DEFAULT_TABLE table=default.Table_28c626ea06e84dc98ac97225be580b7e [id=3c58eaab49fa42f185e2f55d9a3ad95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.352098 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84412b28b4c41c2a3ce18702403dcd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.353430 4504 tablet_service.cc:1467] Processing CreateTablet for tablet 061742d380b1496bbfaaa45ead1d0214 (DEFAULT_TABLE table=default.Table_0d42c9ff3ee949f09e9bd50c70d99727 [id=0adf25c7664142c2b16d63f77b360c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.353775 4504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061742d380b1496bbfaaa45ead1d0214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.354456 4506 tablet_service.cc:1467] Processing CreateTablet for tablet c5c5b08b5efc46cabfd73aaf73b34088 (DEFAULT_TABLE table=default.Table_2ea009902d854d4096d91c7e2d54a289 [id=88e9b126f30e452fb3ce5dc5ae3650bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.354892 4492 tablet_service.cc:1467] Processing CreateTablet for tablet ad383b84ad1646108cc54f1b7ef148ad (DEFAULT_TABLE table=default.Table_66c2a4cc5ab648c5938e6aecd6792779 [id=85d75b729abb421ca3943730e9bb18ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.355324 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8656a886564f06931039b516162b51 (DEFAULT_TABLE table=default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b [id=f4b10783625e48ad95d347379e85460f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:36.356247 5103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.040s sys 0.094s
W20240503 02:12:36.356330 5103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.827s user 0.040s sys 0.094s
I20240503 02:12:36.356671 5103 ts_tablet_manager.cc:1398] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.829s user 0.040s sys 0.096s
W20240503 02:12:36.356719 5103 ts_tablet_manager.cc:1430] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Tablet startup took 828ms
W20240503 02:12:36.356690 5105 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.162s sys 0.228s
W20240503 02:12:36.356801 5105 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.828s user 0.162s sys 0.228s
W20240503 02:12:36.356751 5103 ts_tablet_manager.cc:1432] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Trace:
0503 02:12:20.861505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:20.861574 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:20.862194 (+ 620us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:20.862231 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:20.868219 (+ 5988us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.955137 (+ 86918us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.955499 (+ 362us) inbound_call.cc:173] Queueing success response
0503 02:12:35.511881 (+14556382us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:35.527805 (+ 15924us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:35.527957 (+ 152us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:35.528095 (+ 138us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":27720,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":59,"tablet-open.queue_time_us":14556481,"thread_start_us":826834,"threads_started":1}
I20240503 02:12:36.357290 5103 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:36.357832 5105 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:36.357995 5105 ts_tablet_manager.cc:1367] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.831s user 0.164s sys 0.228s
I20240503 02:12:36.358772 5103 tablet_bootstrap.cc:654] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.359027 5105 raft_consensus.cc:348] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:36.359397 5105 raft_consensus.cc:374] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.359587 5105 raft_consensus.cc:729] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.359872 5105 consensus_queue.cc:260] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:36.360723 5105 ts_tablet_manager.cc:1398] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:12:36.361137 5105 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:36.361723 5114 ts_tablet_manager.cc:1875] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.366003 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86a6bb791449b5a4c490663d87afd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.367280 5103 tablet_bootstrap.cc:492] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:36.367417 5103 ts_tablet_manager.cc:1367] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 02:12:36.368187 5103 raft_consensus.cc:348] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.368379 5103 raft_consensus.cc:374] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.368422 5103 raft_consensus.cc:729] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.368620 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c5b08b5efc46cabfd73aaf73b34088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.369143 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 (DEFAULT_TABLE table=default.Table_5904b0f9c29f483eab4730665a8bb2fd [id=51ad12aaede24d938edb1f16cf568f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.369474 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f3232d509f4cb39eb8923fe1d120a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.375339 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad383b84ad1646108cc54f1b7ef148ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.378031 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet b080fc5ffc564420a0b7f2b7460ebbbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.381213 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 13e31219b2134fa080572238ff0be424 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.381775 4502 tablet_service.cc:1467] Processing CreateTablet for tablet b94c407d27984998a550bef0d6f48c33 (DEFAULT_TABLE table=default.Table_72c760ead308416095fdc381a2737426 [id=5dc985f1f8e74f13ad768c4b0ad9fa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.382351 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.382364 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.382782 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.383091 5828 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.368615 5103 consensus_queue.cc:260] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.383342 6172 raft_consensus.cc:482] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.383456 6172 raft_consensus.cc:504] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.383744 5828 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:36.383908 4808 consensus_meta.cc:287] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Time spent flushing consensus metadata: real 0.837s user 0.003s sys 0.000s
I20240503 02:12:36.384790 6172 leader_election.cc:288] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.385497 5103 ts_tablet_manager.cc:1398] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:12:36.379062 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8656a886564f06931039b516162b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.476478 5103 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:36.423915 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 73252.1us
I20240503 02:12:36.477051 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73198ec769a9486da3f85e8fcbef8e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.477815 5103 tablet_bootstrap.cc:654] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.478392 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 617ad14927c1441b800a204aefc2e619 (table default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]) successfully deleted
W20240503 02:12:36.478631 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 17a24b5c83b649e79644b08253d851d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a24b5c83b649e79644b08253d851d6 already in progress: creating tablet
W20240503 02:12:36.478812 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 552dad16f30a410280c87a6c5ef0eaa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552dad16f30a410280c87a6c5ef0eaa4 already in progress: creating tablet
W20240503 02:12:36.478982 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 50a43e428b804729adf9d8a3133c90ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50a43e428b804729adf9d8a3133c90ff already in progress: creating tablet
W20240503 02:12:36.479162 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4c0745a55223475893edd01d904a223b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0745a55223475893edd01d904a223b already in progress: creating tablet
W20240503 02:12:36.479378 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0700d4aed530413983381d1dc4f78ef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700d4aed530413983381d1dc4f78ef9 already in progress: creating tablet
W20240503 02:12:36.479596 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 46005e39ab284da6bf1bb9565318a2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46005e39ab284da6bf1bb9565318a2f1 already in progress: creating tablet
W20240503 02:12:36.479804 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 818788a24cd1419980c0d622fb05683b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818788a24cd1419980c0d622fb05683b already in progress: creating tablet
W20240503 02:12:36.480015 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9327a9e5472b4eda854320c3a95f8e56 because the tablet was not found. No further retry: Not found: Tablet not found: 9327a9e5472b4eda854320c3a95f8e56
I20240503 02:12:36.480271 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 2f053537029a44d5a519b612d09d19a1 (table default.Table_26db706b59c346f59cb5df08c11c1d22 [id=d9011705ac294bd5832490cfdb4582b9]) successfully deleted
I20240503 02:12:36.480434 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 36a14c4f785f4514ae74b8a0b3073503 (table default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]) successfully deleted
I20240503 02:12:36.384665 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.419816 4639 tablet_service.cc:1467] Processing CreateTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 (DEFAULT_TABLE table=default.Table_28c626ea06e84dc98ac97225be580b7e [id=3c58eaab49fa42f185e2f55d9a3ad95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.481065 4665 raft_consensus.cc:2454] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:36.383992 4808 raft_consensus.cc:2454] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:36.481274 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84412b28b4c41c2a3ce18702403dcd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.481443 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.66:37033 (request call id 139) took 1035 ms. Trace:
I20240503 02:12:36.481545 4808 rpcz_store.cc:276] 0503 02:12:35.445619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.467000 (+ 21381us) service_pool.cc:225] Handling call
0503 02:12:36.481382 (+1014382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:36.483815 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94c407d27984998a550bef0d6f48c33. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:36.384073 4746 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.484390 4462 leader_election.cc:302] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:36.423061 4499 tablet_service.cc:1467] Processing CreateTablet for tablet f5e305683d70467bacf2c6766e0a61b0 (DEFAULT_TABLE table=default.Table_04087507235042c2831d1475b54a2eb8 [id=524f11367c0c4d0299df57099ed43112]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:36.484462 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 179009us
I20240503 02:12:36.484642 6172 raft_consensus.cc:2781] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.484700 6172 raft_consensus.cc:482] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.484733 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e305683d70467bacf2c6766e0a61b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.484751 6172 raft_consensus.cc:3037] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:36.484715 4746 leader_election.cc:341] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:36.484903 4746 leader_election.cc:341] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.486573 4746 leader_election.cc:302] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:36.486971 4746 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.487124 4746 leader_election.cc:302] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.487181 6177 consensus_queue.cc:1035] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.487474 4746 leader_election.cc:302] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.422015 4608 leader_election.cc:302] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:36.487995 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 65996.8us
W20240503 02:12:36.488310 4608 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:36.488373 5851 raft_consensus.cc:2781] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.488425 4608 leader_election.cc:302] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.488472 5851 raft_consensus.cc:482] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.488484 6177 raft_consensus.cc:2781] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.488544 5851 raft_consensus.cc:3037] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.488566 6177 raft_consensus.cc:482] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.488665 6177 raft_consensus.cc:3037] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.428555 4655 tablet_service.cc:1467] Processing CreateTablet for tablet ad383b84ad1646108cc54f1b7ef148ad (DEFAULT_TABLE table=default.Table_66c2a4cc5ab648c5938e6aecd6792779 [id=85d75b729abb421ca3943730e9bb18ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.489701 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad383b84ad1646108cc54f1b7ef148ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.384248 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.490492 4792 tablet_service.cc:1467] Processing CreateTablet for tablet f5e305683d70467bacf2c6766e0a61b0 (DEFAULT_TABLE table=default.Table_04087507235042c2831d1475b54a2eb8 [id=524f11367c0c4d0299df57099ed43112]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.424129 5868 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:36.471130 6041 consensus_queue.cc:1035] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.491050 5990 raft_consensus.cc:2726] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:36.491124 6079 raft_consensus.cc:2781] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.491242 6079 raft_consensus.cc:482] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.491333 6079 raft_consensus.cc:3037] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.388653 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9359d8faab497c86d958f7ec9506de (DEFAULT_TABLE table=default.Table_366708ec2da84f23933deff6182b9fe5 [id=0d8b346069de44f5bc7dfbd98ee5a7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.492166 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.492712 5999 consensus_queue.cc:1035] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.493013 6024 raft_consensus.cc:3037] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.429031 5115 ts_tablet_manager.cc:1888] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.493602 5115 log.cc:1196] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/19fb0929bd234788b5313d96d2e4845c
I20240503 02:12:36.494005 5115 ts_tablet_manager.cc:1909] T 19fb0929bd234788b5313d96d2e4845c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:36.388808 4791 tablet_service.cc:1467] Processing CreateTablet for tablet cd915eae241448f2abe14b04640174b0 (DEFAULT_TABLE table=default.Table_44989b3752354e3e96f57a2f15a27ec6 [id=ddf91007d86046d4b6d6db7a47bb8e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.468992 5106 tablet_bootstrap.cc:654] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.494433 5103 tablet_bootstrap.cc:492] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:36.494607 5103 ts_tablet_manager.cc:1367] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240503 02:12:36.494870 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.495177 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.495605 5103 raft_consensus.cc:348] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.495860 5103 raft_consensus.cc:374] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.495921 5103 raft_consensus.cc:729] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.389031 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.496140 5103 consensus_queue.cc:260] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.496687 4787 tablet_service.cc:1467] Processing CreateTablet for tablet b94c407d27984998a550bef0d6f48c33 (DEFAULT_TABLE table=default.Table_72c760ead308416095fdc381a2737426 [id=5dc985f1f8e74f13ad768c4b0ad9fa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.496889 5103 ts_tablet_manager.cc:1398] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:36.389361 4783 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bba49712445db6d784b6ed94dd7e (DEFAULT_TABLE table=default.Table_94e778467d034214b379477ebfe9694e [id=e42001ef694841f780796904905e3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.497247 5103 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:36.499145 5103 tablet_bootstrap.cc:654] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.389400 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 151f925dcb304594b60c85d850b83533 (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.389429 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 92126108432e4450ba30cfde8ca8618f (DEFAULT_TABLE table=default.Table_a9135679f7fb423cb1d8d69c3430feb0 [id=e1500edee3984ef88898d6fac0f50de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.389748 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 52f167be4c1c43909eacaf5421882f7f (DEFAULT_TABLE table=default.Table_d64beb8ee02743139edc02931b1f97ee [id=c164713a85064ac0b6099b40f3a32473]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.389958 4781 tablet_service.cc:1467] Processing CreateTablet for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b (DEFAULT_TABLE table=default.Table_91951087060348328675a5350c9a1dea [id=4706d758b1094d6081bf011c9415c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.390204 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 0db84f8ed09342ea8b1c3b00b2036c50 (DEFAULT_TABLE table=default.Table_acdeb2544a5243f184185b6c97b607f8 [id=c88bf7514ead43e68bfe83e7cdbfe730]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.390415 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 5e534255f170416690c18ca6bd571117 (DEFAULT_TABLE table=default.Table_b1103caea8f04d32ba60345676c5fa63 [id=d11bb162d01f44f5afab631c1ede7562]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.390436 4775 tablet_service.cc:1467] Processing CreateTablet for tablet d3b9341be62a4275ad6e4c7d7b745621 (DEFAULT_TABLE table=default.Table_1c52b1bbf50148c5b4753da4f8fe72fc [id=c65010247bfd4c7ab61e95464f125c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.390790 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8656a886564f06931039b516162b51 (DEFAULT_TABLE table=default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b [id=f4b10783625e48ad95d347379e85460f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.391252 4785 tablet_service.cc:1467] Processing CreateTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 (DEFAULT_TABLE table=default.Table_28c626ea06e84dc98ac97225be580b7e [id=3c58eaab49fa42f185e2f55d9a3ad95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.393066 5828 raft_consensus.cc:482] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.404636 5105 tablet_bootstrap.cc:654] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.501454 5828 raft_consensus.cc:504] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:36.428664 4811 raft_consensus.cc:2454] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:36.502357 5828 leader_election.cc:288] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:36.502382 6041 raft_consensus.cc:482] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.502499 6041 raft_consensus.cc:504] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.432962 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.433712 4793 tablet_service.cc:1467] Processing CreateTablet for tablet ad383b84ad1646108cc54f1b7ef148ad (DEFAULT_TABLE table=default.Table_66c2a4cc5ab648c5938e6aecd6792779 [id=85d75b729abb421ca3943730e9bb18ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.502852 4746 leader_election.cc:302] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.440181 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 (DEFAULT_TABLE table=default.Table_5904b0f9c29f483eab4730665a8bb2fd [id=51ad12aaede24d938edb1f16cf568f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.709952 5937 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.124:46689
I20240503 02:12:36.440467 4774 tablet_service.cc:1467] Processing CreateTablet for tablet c5c5b08b5efc46cabfd73aaf73b34088 (DEFAULT_TABLE table=default.Table_2ea009902d854d4096d91c7e2d54a289 [id=88e9b126f30e452fb3ce5dc5ae3650bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:36.544485 4260 consensus_peers.cc:485] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d -> Peer 145afaa7548f4b7fafb23f20276f0cb4 (127.4.17.124:46689): Couldn't send request to peer 145afaa7548f4b7fafb23f20276f0cb4. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.124:46689: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:12:36.721006 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 5937
W20240503 02:12:36.847681 4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.124:46689, user_credentials={real_user=slave}} blocked reactor thread for 303206us
I20240503 02:12:36.718178 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.124:46689 every 8 connection(s)
I20240503 02:12:36.503131 6041 leader_election.cc:288] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.506520 4665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.454288 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:36.455799 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:36.590790 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07225ae9ad3c4e44ac0c5c51421b0115 consensus state has no leader
I20240503 02:12:36.848874 4807 raft_consensus.cc:2454] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:36.849092 4266 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 860) took 1071 ms. Trace:
I20240503 02:12:36.436724 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 061742d380b1496bbfaaa45ead1d0214 (DEFAULT_TABLE table=default.Table_0d42c9ff3ee949f09e9bd50c70d99727 [id=0adf25c7664142c2b16d63f77b360c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.518754 6177 raft_consensus.cc:504] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.849202 4266 rpcz_store.cc:276] 0503 02:12:35.777219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.876503 (+ 99284us) service_pool.cc:225] Handling call
0503 02:12:35.877253 (+ 750us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:35.877568 (+ 315us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:35.877874 (+ 306us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:35.877975 (+ 101us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:36.590486 (+712511us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:36.849057 (+258571us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:35.989214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:35.989418 (+ 204us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:35.989429 (+ 11us) write_op.cc:435] Acquired schema lock
0503 02:12:35.989434 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:35.989594 (+ 160us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:35.989615 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:35.989624 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:35.989724 (+ 100us) tablet.cc:699] Row locks acquired
0503 02:12:35.989728 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:35.989779 (+ 51us) write_op.cc:270] Start()
0503 02:12:35.989827 (+ 48us) write_op.cc:276] Timestamp: P: 1714702355989741 usec, L: 0
0503 02:12:35.989833 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.019502 (+ 29669us) log.cc:842] Serialized 405 byte log entry
0503 02:12:36.019793 (+ 291us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:36.574432 (+554639us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.588871 (+ 14439us) write_op.cc:301] APPLY: starting
0503 02:12:36.588959 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.589098 (+ 139us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.589104 (+ 6us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.589271 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.589407 (+ 136us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:36.589458 (+ 51us) write_op.cc:312] APPLY: finished
0503 02:12:36.589698 (+ 240us) log.cc:842] Serialized 23 byte log entry
0503 02:12:36.590202 (+ 504us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.590280 (+ 78us) write_op.cc:454] Released schema lock
0503 02:12:36.590291 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.591073 (+ 782us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.591081 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14277,"apply.run_cpu_time_us":1034,"apply.run_wall_time_us":2415,"num_ops":1,"prepare.queue_time_us":110595,"prepare.run_cpu_time_us":2333,"prepare.run_wall_time_us":30687,"raft.queue_time_us":49,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":9,"replication_time_us":584563,"spinlock_wait_cycles":67548160,"thread_start_us":13369,"threads_started":1}]]}
I20240503 02:12:36.849788 6177 leader_election.cc:288] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.519625 5159 ts_tablet_manager.cc:1888] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:36.849987 5159 log.cc:1196] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3b41dce88919456aa846dbdcf5e2759f
I20240503 02:12:36.850086 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_28c626ea06e84dc98ac97225be580b7e" } modify_external_catalogs: true
I20240503 02:12:36.850291 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_28c626ea06e84dc98ac97225be580b7e" } modify_external_catalogs: true
I20240503 02:12:36.850359 5159 ts_tablet_manager.cc:1909] T 3b41dce88919456aa846dbdcf5e2759f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:36.851248 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 572) took 3631 ms. Trace:
I20240503 02:12:36.720084 5937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
I20240503 02:12:36.851341 5159 rpcz_store.cc:276] 0503 02:12:33.219664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.220111 (+ 447us) service_pool.cc:225] Handling call
0503 02:12:36.331789 (+3111678us) ts_tablet_manager.cc:1120] Deleting tablet 3b41dce88919456aa846dbdcf5e2759f
0503 02:12:36.331800 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.428879 (+ 97079us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.519480 (+ 90601us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.851224 (+331744us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3111496}
I20240503 02:12:36.851747 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 573) took 3631 ms. Trace:
I20240503 02:12:36.851820 5159 rpcz_store.cc:276] 0503 02:12:33.219793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.220322 (+ 529us) service_pool.cc:225] Handling call
0503 02:12:36.851638 (+3631316us) ts_tablet_manager.cc:1120] Deleting tablet 37a54873b48847c39a2b961ab5c3806c
0503 02:12:36.851649 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.851732 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3631124}
I20240503 02:12:36.504462 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.852108 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 575) took 3632 ms. Trace:
I20240503 02:12:36.852176 5159 rpcz_store.cc:276] 0503 02:12:33.220063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.220542 (+ 479us) service_pool.cc:225] Handling call
0503 02:12:36.852036 (+3631494us) ts_tablet_manager.cc:1120] Deleting tablet dedead5a2bf347f6991cce8e9b4db708
0503 02:12:36.852046 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.852093 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3631311}
I20240503 02:12:36.514628 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09005fb01c384122a7310c1eea12d3f0" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.852411 5159 tablet_replica.cc:330] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:36.852432 4674 raft_consensus.cc:2454] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:36.852638 5159 raft_consensus.cc:2229] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.593387 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7178760c21904207b6c8317093871f63
I20240503 02:12:36.852805 5159 raft_consensus.cc:2258] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.521307 6079 raft_consensus.cc:504] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.853268 4275 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 858) took 1086 ms. Trace:
I20240503 02:12:36.853458 6079 leader_election.cc:288] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.522639 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad383b84ad1646108cc54f1b7ef148ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.524421 5828 raft_consensus.cc:2781] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.853706 5828 raft_consensus.cc:686] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:36.853376 4275 rpcz_store.cc:276] 0503 02:12:35.766377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.766647 (+ 270us) service_pool.cc:225] Handling call
0503 02:12:35.767238 (+ 591us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:35.767440 (+ 202us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:35.767523 (+ 83us) catalog_manager.cc:2758] Locking tablets
0503 02:12:35.984910 (+217387us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:36.593079 (+608169us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:36.593207 (+ 128us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:36.593275 (+ 68us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:36.853234 (+259959us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.019940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.020104 (+ 164us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.020114 (+ 10us) write_op.cc:435] Acquired schema lock
0503 02:12:36.020119 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:36.020302 (+ 183us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.020318 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.020326 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.020478 (+ 152us) tablet.cc:699] Row locks acquired
0503 02:12:36.020483 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:36.020528 (+ 45us) write_op.cc:270] Start()
0503 02:12:36.020568 (+ 40us) write_op.cc:276] Timestamp: P: 1714702356020486 usec, L: 0
0503 02:12:36.020574 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.020790 (+ 216us) log.cc:842] Serialized 756 byte log entry
0503 02:12:36.587965 (+567175us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.591408 (+ 3443us) write_op.cc:301] APPLY: starting
0503 02:12:36.591428 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.591559 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.591649 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.591866 (+ 217us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.591996 (+ 130us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:36.592009 (+ 13us) write_op.cc:312] APPLY: finished
0503 02:12:36.592365 (+ 356us) log.cc:842] Serialized 29 byte log entry
0503 02:12:36.592786 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.592847 (+ 61us) write_op.cc:454] Released schema lock
0503 02:12:36.592854 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.593054 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.593060 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3305,"apply.run_cpu_time_us":963,"apply.run_wall_time_us":1830,"mutex_wait_us":3451,"num_ops":2,"prepare.queue_time_us":34189,"prepare.run_cpu_time_us":969,"prepare.run_wall_time_us":967,"raft.queue_time_us":4606,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":567359,"thread_start_us":144878,"threads_started":1}]]}
I20240503 02:12:36.853924 5828 consensus_queue.cc:237] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.854156 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 8135907b78284d7ca74396cc77dd0612 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:36.854753 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7eb0b6747a1a4599bfa9c6121b1c7c32" } new_table_name: "default.Table_6518e9cc40e1483f885638cd1a931f6a" modify_external_catalogs: true
I20240503 02:12:36.854964 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73198ec769a9486da3f85e8fcbef8e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.504753 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:36.856504 4527 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 2.
I20240503 02:12:36.517958 5851 raft_consensus.cc:504] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.520714 6024 raft_consensus.cc:2726] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:36.522720 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 560) took 3497 ms. Trace:
I20240503 02:12:36.524648 4746 leader_election.cc:302] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.856853 5115 rpcz_store.cc:276] 0503 02:12:33.025355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.036775 (+ 11420us) service_pool.cc:225] Handling call
0503 02:12:35.723572 (+2686797us) ts_tablet_manager.cc:1120] Deleting tablet 19fb0929bd234788b5313d96d2e4845c
0503 02:12:35.723583 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.304735 (+581152us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.428867 (+124132us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.522693 (+ 93826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2686423}
W20240503 02:12:36.857185 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 332544us
I20240503 02:12:36.857215 5851 leader_election.cc:288] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:36.734127 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5cd296bb3b574d6cbb0252ad10ada00c
I20240503 02:12:36.857347 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 562) took 3819 ms. Trace:
I20240503 02:12:36.857414 5115 rpcz_store.cc:276] 0503 02:12:33.038320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.039122 (+ 802us) service_pool.cc:225] Handling call
0503 02:12:36.857203 (+3818081us) ts_tablet_manager.cc:1120] Deleting tablet 4c0745a55223475893edd01d904a223b
0503 02:12:36.857214 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.857328 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3816655}
I20240503 02:12:36.857671 5115 tablet_replica.cc:330] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:36.857671 4746 leader_election.cc:341] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.857772 4272 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 855) took 1119 ms. Trace:
I20240503 02:12:36.857887 5115 raft_consensus.cc:2229] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:36.858050 5115 raft_consensus.cc:2258] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.506196 5103 tablet_bootstrap.cc:492] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:36.858305 5103 ts_tablet_manager.cc:1367] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.361s user 0.002s sys 0.005s
I20240503 02:12:36.857877 4272 rpcz_store.cc:276] 0503 02:12:35.738102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.738152 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:35.738727 (+ 575us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:35.738935 (+ 208us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:35.739031 (+ 96us) catalog_manager.cc:2758] Locking tablets
0503 02:12:35.984761 (+245730us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:36.733830 (+749069us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:36.733953 (+ 123us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:36.734021 (+ 68us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:36.857732 (+123711us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.020986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.021105 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.021112 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.021114 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:36.021218 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.021245 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.021252 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.021384 (+ 132us) tablet.cc:699] Row locks acquired
0503 02:12:36.021388 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:36.021426 (+ 38us) write_op.cc:270] Start()
0503 02:12:36.021462 (+ 36us) write_op.cc:276] Timestamp: P: 1714702356021388 usec, L: 0
0503 02:12:36.021465 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.021642 (+ 177us) log.cc:842] Serialized 756 byte log entry
0503 02:12:36.733035 (+711393us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.733158 (+ 123us) write_op.cc:301] APPLY: starting
0503 02:12:36.733183 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.733244 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.733283 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.733389 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.733442 (+ 53us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:36.733458 (+ 16us) write_op.cc:312] APPLY: finished
0503 02:12:36.733567 (+ 109us) log.cc:842] Serialized 29 byte log entry
0503 02:12:36.733649 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.733679 (+ 30us) write_op.cc:454] Released schema lock
0503 02:12:36.733689 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.733765 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.733768 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":676,"apply.run_wall_time_us":677,"mutex_wait_us":3441,"num_ops":2,"prepare.queue_time_us":35154,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":757,"raft.queue_time_us":3786,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":711532}]]}
I20240503 02:12:36.858598 5115 ts_tablet_manager.cc:1875] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.858729 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c77b55b8023b459c96d97e8d9c8f7986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:36.858978 5103 raft_consensus.cc:348] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.859148 5103 raft_consensus.cc:374] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.859197 5103 raft_consensus.cc:729] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.859289 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f3232d509f4cb39eb8923fe1d120a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:36.859297 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c77b55b8023b459c96d97e8d9c8f7986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.859377 5103 consensus_queue.cc:260] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.859915 5103 ts_tablet_manager.cc:1398] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:36.860206 5103 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:36.860453 6078 raft_consensus.cc:2781] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.860548 6078 raft_consensus.cc:686] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:36.734571 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6d08e7d17f1644749a9a2e7175b3191a
I20240503 02:12:36.860751 6078 consensus_queue.cc:237] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.861202 6174 consensus_queue.cc:1035] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.861369 4269 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 861) took 1084 ms. Trace:
I20240503 02:12:36.861696 6174 consensus_queue.cc:1035] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.861461 4269 rpcz_store.cc:276] 0503 02:12:35.777302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.983080 (+205778us) service_pool.cc:225] Handling call
0503 02:12:36.012892 (+ 29812us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:36.013341 (+ 449us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:36.013461 (+ 120us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:36.013536 (+ 75us) catalog_manager.cc:2758] Locking tablets
0503 02:12:36.013683 (+ 147us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:36.734364 (+720681us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:36.734456 (+ 92us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:36.734506 (+ 50us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:36.861318 (+126812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.021782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.021899 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.021907 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:36.021911 (+ 4us) tablet.cc:660] Decoding operations
0503 02:12:36.022039 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.022051 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.022059 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.022167 (+ 108us) tablet.cc:699] Row locks acquired
0503 02:12:36.022172 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:36.022204 (+ 32us) write_op.cc:270] Start()
0503 02:12:36.022235 (+ 31us) write_op.cc:276] Timestamp: P: 1714702356022168 usec, L: 0
0503 02:12:36.022241 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.022460 (+ 219us) log.cc:842] Serialized 725 byte log entry
0503 02:12:36.733136 (+710676us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.733874 (+ 738us) write_op.cc:301] APPLY: starting
0503 02:12:36.733890 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.733930 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.733968 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.734038 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.734088 (+ 50us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:36.734098 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:36.734167 (+ 69us) log.cc:842] Serialized 29 byte log entry
0503 02:12:36.734227 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.734242 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:36.734250 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.734305 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.734310 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":67929984,"child_traces":[["op",{"apply.queue_time_us":700,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":486,"num_ops":2,"prepare.queue_time_us":7425,"prepare.run_cpu_time_us":791,"prepare.run_wall_time_us":790,"raft.queue_time_us":2978,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":5,"replication_time_us":710877}]]}
I20240503 02:12:36.862010 6174 consensus_queue.cc:1035] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.862079 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet 1436dd3df7284f9bb379236eb9d76836 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:36.862545 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet 6b15997a94b74ad3808d965acfb5d155 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:36.738574 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0a8c57ddf3d8428e8fa5ffb061dc054b
I20240503 02:12:36.507617 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e31219b2134fa080572238ff0be424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.863643 4271 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 864) took 1073 ms. Trace:
W20240503 02:12:36.537043 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 48279.8us
I20240503 02:12:36.863986 4608 leader_election.cc:302] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:36.863734 4271 rpcz_store.cc:276] 0503 02:12:35.790232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.983774 (+193542us) service_pool.cc:225] Handling call
0503 02:12:36.017668 (+ 33894us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:36.018008 (+ 340us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:36.018158 (+ 150us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:36.018231 (+ 73us) catalog_manager.cc:2758] Locking tablets
0503 02:12:36.018377 (+ 146us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:36.738414 (+720037us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:36.738484 (+ 70us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:36.738517 (+ 33us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:36.863597 (+125080us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.027734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.027837 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.027844 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.027847 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:36.027960 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.027970 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.027976 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.028134 (+ 158us) tablet.cc:699] Row locks acquired
0503 02:12:36.028138 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:36.028171 (+ 33us) write_op.cc:270] Start()
0503 02:12:36.028201 (+ 30us) write_op.cc:276] Timestamp: P: 1714702356028135 usec, L: 0
0503 02:12:36.028205 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.028376 (+ 171us) log.cc:842] Serialized 725 byte log entry
0503 02:12:36.737381 (+709005us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.738065 (+ 684us) write_op.cc:301] APPLY: starting
0503 02:12:36.738076 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.738102 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.738135 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.738183 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.738228 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:36.738235 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:36.738274 (+ 39us) log.cc:842] Serialized 29 byte log entry
0503 02:12:36.738306 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.738316 (+ 10us) write_op.cc:454] Released schema lock
0503 02:12:36.738323 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.738374 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.738378 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":77444352,"child_traces":[["op",{"apply.queue_time_us":649,"apply.run_cpu_time_us":357,"apply.run_wall_time_us":356,"num_ops":2,"prepare.queue_time_us":8841,"prepare.run_cpu_time_us":745,"prepare.run_wall_time_us":1143,"raft.queue_time_us":32,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":709159}]]}
I20240503 02:12:36.864456 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 0af50eb2b1c949a0a6a71e64a894e8a5 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:36.561679 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 13e31219b2134fa080572238ff0be424 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.519155 6172 raft_consensus.cc:504] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.875463 6172 leader_election.cc:288] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.524937 6013 raft_consensus.cc:482] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.875665 6013 raft_consensus.cc:504] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.876334 6013 leader_election.cc:288] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.613886 5868 raft_consensus.cc:504] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.876788 5868 leader_election.cc:288] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 3 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:36.507406 4462 leader_election.cc:302] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:36.877234 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 369824us
I20240503 02:12:36.867765 5114 ts_tablet_manager.cc:1888] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:36.879331 5159 ts_tablet_manager.cc:1875] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.862756 6166 consensus_queue.cc:1035] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.863915 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f167be4c1c43909eacaf5421882f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.537986 5105 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:36.880645 5105 ts_tablet_manager.cc:1367] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.520s user 0.005s sys 0.002s
I20240503 02:12:36.881527 5105 raft_consensus.cc:348] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:36.881717 5105 raft_consensus.cc:374] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.881775 5105 raft_consensus.cc:729] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.881974 5105 consensus_queue.cc:260] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:36.882220 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9359d8faab497c86d958f7ec9506de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.882495 5105 ts_tablet_manager.cc:1398] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:36.882762 5105 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:36.884133 5105 tablet_bootstrap.cc:654] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.885182 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38cb5fc65ef4ad2b24c9351f36f6d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.884219 5103 tablet_bootstrap.cc:654] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.889102 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b9341be62a4275ad6e4c7d7b745621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.893069 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84412b28b4c41c2a3ce18702403dcd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.893795 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8656a886564f06931039b516162b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.897039 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd915eae241448f2abe14b04640174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.901060 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bba49712445db6d784b6ed94dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.902547 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db84f8ed09342ea8b1c3b00b2036c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.904976 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061742d380b1496bbfaaa45ead1d0214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.905429 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94c407d27984998a550bef0d6f48c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.906494 5159 ts_tablet_manager.cc:1888] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.906723 5159 log.cc:1196] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8750b02c5936412db525d2d3e77befeb
I20240503 02:12:36.907055 5159 ts_tablet_manager.cc:1909] T 8750b02c5936412db525d2d3e77befeb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:36.907789 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 571) took 3688 ms. Trace:
I20240503 02:12:36.907868 5159 rpcz_store.cc:276] 0503 02:12:33.219518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.220756 (+ 1238us) service_pool.cc:225] Handling call
0503 02:12:36.852364 (+3631608us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:36.852373 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.891763 (+ 39390us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.906401 (+ 14638us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.907765 (+ 1364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3631479}
I20240503 02:12:36.908308 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 576) took 3630 ms. Trace:
I20240503 02:12:36.908377 5159 rpcz_store.cc:276] 0503 02:12:33.278302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.278371 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:36.908182 (+3629811us) ts_tablet_manager.cc:1120] Deleting tablet d35138f04f224475af324e77aa1e0191
0503 02:12:36.908192 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.908292 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3629575}
I20240503 02:12:36.908669 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 577) took 3594 ms. Trace:
I20240503 02:12:36.908731 5159 rpcz_store.cc:276] 0503 02:12:33.313958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.314023 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:36.908583 (+3594560us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:36.908590 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.908657 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594339}
I20240503 02:12:36.848829 4806 raft_consensus.cc:2454] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:36.909358 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e534255f170416690c18ca6bd571117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.910135 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151f925dcb304594b60c85d850b83533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.913064 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e31219b2134fa080572238ff0be424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.865257 6262 raft_consensus.cc:2781] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.913666 6262 raft_consensus.cc:482] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.913806 6262 raft_consensus.cc:3037] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.914381 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c5b08b5efc46cabfd73aaf73b34088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.914918 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92126108432e4450ba30cfde8ca8618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.865460 5999 consensus_queue.cc:1035] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.545445 5106 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:36.917104 5106 ts_tablet_manager.cc:1367] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.616s user 0.010s sys 0.004s
I20240503 02:12:36.917670 5868 raft_consensus.cc:3037] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.917835 5106 raft_consensus.cc:348] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.918007 5106 raft_consensus.cc:374] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.918053 5106 raft_consensus.cc:729] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.918241 5106 consensus_queue.cc:260] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.918705 5106 ts_tablet_manager.cc:1398] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:36.918974 5106 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:36.920316 5106 tablet_bootstrap.cc:654] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.920282 5868 raft_consensus.cc:2726] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:36.865981 6247 consensus_queue.cc:1035] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.921113 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e305683d70467bacf2c6766e0a61b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.866531 6024 consensus_queue.cc:1035] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.881613 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:36.881817 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:36.882000 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:36.889708 5115 ts_tablet_manager.cc:1888] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.922602 4667 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Advancing to term 3
I20240503 02:12:36.922610 5115 log.cc:1196] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3e6a3ca705e8482a8cd6a6fb17d93470
I20240503 02:12:36.923022 5115 ts_tablet_manager.cc:1909] T 3e6a3ca705e8482a8cd6a6fb17d93470 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:36.923792 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 559) took 3898 ms. Trace:
I20240503 02:12:36.923871 5115 rpcz_store.cc:276] 0503 02:12:33.025159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.037208 (+ 12049us) service_pool.cc:225] Handling call
0503 02:12:36.857628 (+3820420us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:36.857637 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.867389 (+ 9752us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.889593 (+ 22204us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.923770 (+ 34177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3817032}
I20240503 02:12:36.924253 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 563) took 3723 ms. Trace:
I20240503 02:12:36.924326 5115 rpcz_store.cc:276] 0503 02:12:33.201191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.201683 (+ 492us) service_pool.cc:225] Handling call
0503 02:12:36.924114 (+3722431us) ts_tablet_manager.cc:1120] Deleting tablet 46005e39ab284da6bf1bb9565318a2f1
0503 02:12:36.924123 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.924237 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722191}
I20240503 02:12:36.924631 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 564) took 3708 ms. Trace:
I20240503 02:12:36.924687 5115 rpcz_store.cc:276] 0503 02:12:33.215892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.216605 (+ 713us) service_pool.cc:225] Handling call
0503 02:12:36.924539 (+3707934us) ts_tablet_manager.cc:1120] Deleting tablet 818788a24cd1419980c0d622fb05683b
0503 02:12:36.924547 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.924619 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3707683}
I20240503 02:12:36.924948 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 565) took 3646 ms. Trace:
I20240503 02:12:36.925009 5115 rpcz_store.cc:276] 0503 02:12:33.278852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.278912 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:36.924848 (+3645936us) ts_tablet_manager.cc:1120] Deleting tablet d35138f04f224475af324e77aa1e0191
0503 02:12:36.924855 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.924910 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3645740}
I20240503 02:12:36.925298 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 566) took 3610 ms. Trace:
I20240503 02:12:36.925364 5115 rpcz_store.cc:276] 0503 02:12:33.314473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.314540 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:36.925202 (+3610662us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:36.925211 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.925284 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3610421}
I20240503 02:12:36.925678 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 567) took 3606 ms. Trace:
I20240503 02:12:36.925766 5115 rpcz_store.cc:276] 0503 02:12:33.318919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.318979 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:36.925571 (+3606592us) ts_tablet_manager.cc:1120] Deleting tablet 76aef6b021da4f5aad7f523b15d484ec
0503 02:12:36.925580 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.925664 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3606373}
I20240503 02:12:36.926048 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 568) took 3583 ms. Trace:
I20240503 02:12:36.926115 5115 rpcz_store.cc:276] 0503 02:12:33.342923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.342985 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:36.925952 (+3582967us) ts_tablet_manager.cc:1120] Deleting tablet c757a3a979c2489daa621cf8ea293ea8
0503 02:12:36.925961 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.926033 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3582680}
I20240503 02:12:36.926313 5115 tablet_replica.cc:330] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:36.926533 5115 raft_consensus.cc:2229] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.926677 5115 raft_consensus.cc:2258] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.927225 5115 ts_tablet_manager.cc:1875] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.929162 5114 log.cc:1196] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:36.929622 5114 ts_tablet_manager.cc:1909] T d271d2c112ba44b68d2933a6ebfe7a18 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.930538 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 543) took 5394 ms. Trace:
I20240503 02:12:36.930641 5114 rpcz_store.cc:276] 0503 02:12:31.535635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.070812 (+535177us) service_pool.cc:225] Handling call
0503 02:12:36.329900 (+4259088us) ts_tablet_manager.cc:1120] Deleting tablet d271d2c112ba44b68d2933a6ebfe7a18
0503 02:12:36.329956 (+ 56us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.449335 (+119379us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.866736 (+417401us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.930510 (+ 63774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4258900}
I20240503 02:12:36.931178 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 544) took 5395 ms. Trace:
I20240503 02:12:36.931270 5114 rpcz_store.cc:276] 0503 02:12:31.535754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.070969 (+535215us) service_pool.cc:225] Handling call
0503 02:12:36.931035 (+4860066us) ts_tablet_manager.cc:1120] Deleting tablet fdc2850c07d042a7951a5fbeb0e4945d
0503 02:12:36.931048 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.931160 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4859854}
I20240503 02:12:36.931643 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 545) took 5395 ms. Trace:
I20240503 02:12:36.931723 5114 rpcz_store.cc:276] 0503 02:12:31.535858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.071194 (+535336us) service_pool.cc:225] Handling call
0503 02:12:36.931535 (+4860341us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:36.931545 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.931626 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4860185}
I20240503 02:12:36.931982 5114 tablet_replica.cc:330] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:36.878448 4462 leader_election.cc:302] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:36.932161 5114 raft_consensus.cc:2229] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.932317 5114 raft_consensus.cc:2258] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:36.932250 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 53814.7us
I20240503 02:12:36.934388 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 578) took 3617 ms. Trace:
I20240503 02:12:36.934485 5159 rpcz_store.cc:276] 0503 02:12:33.317015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.317070 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:36.908900 (+3591830us) ts_tablet_manager.cc:1120] Deleting tablet 76aef6b021da4f5aad7f523b15d484ec
0503 02:12:36.908907 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.934365 (+ 25458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3591598}
I20240503 02:12:36.934974 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 580) took 3592 ms. Trace:
I20240503 02:12:36.935042 5159 rpcz_store.cc:276] 0503 02:12:33.342414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.342473 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:36.934870 (+3592397us) ts_tablet_manager.cc:1120] Deleting tablet c757a3a979c2489daa621cf8ea293ea8
0503 02:12:36.934879 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.934958 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3592168}
I20240503 02:12:36.935294 5159 tablet_replica.cc:330] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:36.878612 4525 raft_consensus.cc:2354] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 for earlier term 2. Current term is 3.
I20240503 02:12:36.935475 5159 raft_consensus.cc:2229] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.937652 5159 raft_consensus.cc:2258] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.938153 5159 ts_tablet_manager.cc:1875] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.864884 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet aa67857b858c4719b498291c52b7aceb from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:36.932888 5114 ts_tablet_manager.cc:1875] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.934229 5103 tablet_bootstrap.cc:492] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:36.940240 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f8cf5f90991c472a9fb961b4e6d6829e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:36.940517 5103 ts_tablet_manager.cc:1367] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.005s
W20240503 02:12:36.941010 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8cf5f90991c472a9fb961b4e6d6829e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.941247 5103 raft_consensus.cc:348] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.941437 5103 raft_consensus.cc:374] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.941505 5103 raft_consensus.cc:729] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.941682 5103 consensus_queue.cc:260] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.745121 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e800e34461a64783bd120662f37d31a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:36.942287 5103 ts_tablet_manager.cc:1398] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 02:12:36.942471 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e800e34461a64783bd120662f37d31a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.942587 5103 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:36.745986 4267 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.66:56851 (request call id 85) took 848 ms (client timeout 1000 ms). Trace:
I20240503 02:12:36.944707 5115 ts_tablet_manager.cc:1888] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:36.944813 5115 log.cc:1196] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/9327a9e5472b4eda854320c3a95f8e56
W20240503 02:12:36.944308 4267 rpcz_store.cc:269] 0503 02:12:35.897529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.982586 (+ 85057us) service_pool.cc:225] Handling call
0503 02:12:36.019055 (+ 36469us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:36.745918 (+726863us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.028903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.029025 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.029030 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:36.029033 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:36.029117 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.029127 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.029134 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:36.029190 (+ 56us) tablet.cc:699] Row locks acquired
0503 02:12:36.029194 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:36.029225 (+ 31us) write_op.cc:270] Start()
0503 02:12:36.029251 (+ 26us) write_op.cc:276] Timestamp: P: 1714702356029189 usec, L: 0
0503 02:12:36.029254 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.029440 (+ 186us) log.cc:842] Serialized 527 byte log entry
0503 02:12:36.737421 (+707981us) op_driver.cc:444] REPLICATION: finished
0503 02:12:36.738449 (+ 1028us) write_op.cc:301] APPLY: starting
0503 02:12:36.738459 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:36.738484 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:36.738506 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:36.738543 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:36.738576 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:36.738586 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:36.738639 (+ 53us) log.cc:842] Serialized 23 byte log entry
0503 02:12:36.738677 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.738693 (+ 16us) write_op.cc:454] Released schema lock
0503 02:12:36.738699 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:36.738746 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:36.738750 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"spinlock_wait_cycles":81228032,"child_traces":[["op",{"apply.queue_time_us":1001,"apply.run_cpu_time_us":339,"apply.run_wall_time_us":339,"num_ops":1,"prepare.queue_time_us":2591,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":1048,"raft.queue_time_us":40,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":708155,"thread_start_us":3315,"threads_started":1}]]}
I20240503 02:12:36.945119 5115 ts_tablet_manager.cc:1909] T 9327a9e5472b4eda854320c3a95f8e56 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:36.945413 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7b6f1b8dd48144eb88d5710246516d40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:36.945888 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 570) took 3574 ms. Trace:
W20240503 02:12:36.945941 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b6f1b8dd48144eb88d5710246516d40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:36.945958 5114 ts_tablet_manager.cc:1888] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:36.946053 5114 log.cc:1196] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/09005fb01c384122a7310c1eea12d3f0
I20240503 02:12:36.945981 5115 rpcz_store.cc:276] 0503 02:12:33.371046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.371251 (+ 205us) service_pool.cc:225] Handling call
0503 02:12:36.926278 (+3555027us) ts_tablet_manager.cc:1120] Deleting tablet 9327a9e5472b4eda854320c3a95f8e56
0503 02:12:36.926285 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.930575 (+ 4290us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.944598 (+ 14023us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.945865 (+ 1267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3554727}
I20240503 02:12:36.946298 5115 tablet_replica.cc:330] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:36.946343 5114 ts_tablet_manager.cc:1909] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.946494 5115 raft_consensus.cc:2229] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.946643 5115 raft_consensus.cc:2258] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.946739 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 538) took 5972 ms. Trace:
I20240503 02:12:36.946825 5114 rpcz_store.cc:276] 0503 02:12:30.974154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.038952 (+1064798us) service_pool.cc:225] Handling call
0503 02:12:36.931942 (+4892990us) ts_tablet_manager.cc:1120] Deleting tablet 09005fb01c384122a7310c1eea12d3f0
0503 02:12:36.931949 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.942560 (+ 10611us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.945863 (+ 3303us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.946717 (+ 854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4859405}
W20240503 02:12:36.776151 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 984fe4640980407aa03c16c14a1303c1 (table default.Table_04dad7f94d5f493da9ac2818393cf141 [id=4915cc5437ad47798f1763ece3572049], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 984fe4640980407aa03c16c14a1303c1 consensus state has no leader
I20240503 02:12:36.947150 5114 tablet_replica.cc:330] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:36.947373 5114 raft_consensus.cc:2229] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.947523 5114 raft_consensus.cc:2258] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.948127 5114 ts_tablet_manager.cc:1875] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.948828 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet d271d2c112ba44b68d2933a6ebfe7a18 (table default.Table_d54df163e4634a779d3bcfddb083db60 [id=363231755ce84709b926a7416f3bc40a]) successfully deleted
W20240503 02:12:36.949087 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fdc2850c07d042a7951a5fbeb0e4945d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc2850c07d042a7951a5fbeb0e4945d already in progress: creating tablet
W20240503 02:12:36.949293 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 48e713ee37d042309166b0f118e14986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e713ee37d042309166b0f118e14986 already in progress: creating tablet
I20240503 02:12:36.949436 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 09005fb01c384122a7310c1eea12d3f0 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:36.949774 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3b41dce88919456aa846dbdcf5e2759f (table default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]) successfully deleted
W20240503 02:12:36.949952 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 37a54873b48847c39a2b961ab5c3806c because the tablet was not found. No further retry: Not found: Tablet not found: 37a54873b48847c39a2b961ab5c3806c
W20240503 02:12:36.950110 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet dedead5a2bf347f6991cce8e9b4db708 because the tablet was not found. No further retry: Not found: Tablet not found: dedead5a2bf347f6991cce8e9b4db708
I20240503 02:12:36.950412 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8750b02c5936412db525d2d3e77befeb (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]) successfully deleted
W20240503 02:12:36.950583 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d35138f04f224475af324e77aa1e0191 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35138f04f224475af324e77aa1e0191 already in progress: creating tablet
W20240503 02:12:36.950745 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8195767d80384a6c9a4690292f6776de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8195767d80384a6c9a4690292f6776de already in progress: creating tablet
W20240503 02:12:36.950903 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 76aef6b021da4f5aad7f523b15d484ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76aef6b021da4f5aad7f523b15d484ec already in progress: creating tablet
W20240503 02:12:36.951136 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c757a3a979c2489daa621cf8ea293ea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c757a3a979c2489daa621cf8ea293ea8 already in progress: creating tablet
W20240503 02:12:36.819382 4248 kernel_stack_watchdog.cc:198] Thread 4260 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fd3ea573980 (unknown) at ??:0
@ 0x7fd3e3395255 __write at ??:0
@ 0x7fd3e331028d _IO_file_write at ??:0
@ 0x7fd3e3310bff _IO_file_xsputn at ??:0
@ 0x7fd3e3304a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7fd3e44f78a0 google::LogMessage::SendToLog() at ??:0
@ 0x7fd3e44f37cc google::LogMessage::Flush() at ??:0
@ 0x7fd3e44f3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd3f29309ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7fd3f292f69f kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7fd3f2936061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7fd3f2935e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fd3ede1f4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fd3ede20365 kudu::rpc::OutboundCall::SetFailed() at ??:0
I20240503 02:12:36.952100 5103 tablet_bootstrap.cc:654] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.952517 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.952630 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178760c21904207b6c8317093871f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.744678 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f386f3dc954a401c9210cc2be3307c4d" } new_table_name: "default.Table_db80a3085448432ca09aafe2ba19dc90" modify_external_catalogs: true
I20240503 02:12:36.952749 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.952936 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:36.954699 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:36.935566 4525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.17.66:57571 (request call id 143) took 1391 ms. Trace:
I20240503 02:12:36.954943 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:36.954908 4525 rpcz_store.cc:276] 0503 02:12:35.544301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.692617 (+148316us) service_pool.cc:225] Handling call
0503 02:12:36.935505 (+1242888us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x6160001c1554. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139eb7c6b5 00007f13a284ed24 00007f139cf529b6 00007f139cf526e5 00007f139a02504e 00007f139a023f2e 00007f139a02a405 00007f139a02ad2f 00007f139a02ab58 0000000000a60bcc
0503 02:12:36.935548 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2648582528}
I20240503 02:12:36.955082 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:36.956146 5114 ts_tablet_manager.cc:1888] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.956251 5114 log.cc:1196] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3e6a3ca705e8482a8cd6a6fb17d93470
I20240503 02:12:36.956565 5114 ts_tablet_manager.cc:1909] T 3e6a3ca705e8482a8cd6a6fb17d93470 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.957298 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 546) took 4671 ms. Trace:
I20240503 02:12:36.957381 5114 rpcz_store.cc:276] 0503 02:12:32.287354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.287445 (+ 91us) service_pool.cc:225] Handling call
0503 02:12:36.947108 (+4659663us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:36.947117 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.951861 (+ 4744us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.956051 (+ 4190us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.957277 (+ 1226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4659016}
I20240503 02:12:36.957742 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 547) took 4516 ms. Trace:
I20240503 02:12:36.957800 5114 rpcz_store.cc:276] 0503 02:12:32.440825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.441163 (+ 338us) service_pool.cc:225] Handling call
0503 02:12:36.957629 (+4516466us) ts_tablet_manager.cc:1120] Deleting tablet ec641e403ed843fda31dabd3920aeb68
0503 02:12:36.957639 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.957730 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4516189}
I20240503 02:12:36.958041 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 548) took 4489 ms. Trace:
I20240503 02:12:36.958093 5114 rpcz_store.cc:276] 0503 02:12:32.468343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.468402 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:36.957963 (+4489561us) ts_tablet_manager.cc:1120] Deleting tablet fcfb174825f1441f861ae6816137f855
0503 02:12:36.957970 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.958030 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4489305}
I20240503 02:12:36.958429 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 550) took 4248 ms. Trace:
I20240503 02:12:36.958492 5114 rpcz_store.cc:276] 0503 02:12:32.709741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.709793 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:36.958338 (+4248545us) ts_tablet_manager.cc:1120] Deleting tablet 17a24b5c83b649e79644b08253d851d6
0503 02:12:36.958347 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.958415 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4227169}
I20240503 02:12:36.958743 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 551) took 4241 ms. Trace:
I20240503 02:12:36.958793 5114 rpcz_store.cc:276] 0503 02:12:32.717359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.717448 (+ 89us) service_pool.cc:225] Handling call
0503 02:12:36.958669 (+4241221us) ts_tablet_manager.cc:1120] Deleting tablet 552dad16f30a410280c87a6c5ef0eaa4
0503 02:12:36.958676 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.958733 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4227422}
I20240503 02:12:36.959024 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 552) took 4175 ms. Trace:
I20240503 02:12:36.959079 5114 rpcz_store.cc:276] 0503 02:12:32.784053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.784624 (+ 571us) service_pool.cc:225] Handling call
0503 02:12:36.958947 (+4174323us) ts_tablet_manager.cc:1120] Deleting tablet 50a43e428b804729adf9d8a3133c90ff
0503 02:12:36.958953 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.959013 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4174091}
I20240503 02:12:36.959285 5114 tablet_replica.cc:330] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:36.959506 5114 raft_consensus.cc:2229] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:36.959787 5114 raft_consensus.cc:2258] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.960436 5114 ts_tablet_manager.cc:1875] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.961998 5103 tablet_bootstrap.cc:492] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:36.962116 5103 ts_tablet_manager.cc:1367] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:12:36.962836 5103 raft_consensus.cc:348] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.963018 5103 raft_consensus.cc:374] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.963061 5103 raft_consensus.cc:729] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.963234 5103 consensus_queue.cc:260] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.971150 5159 ts_tablet_manager.cc:1888] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:36.971293 5159 log.cc:1196] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d022b5df4a914e439fff918f70c34157
I20240503 02:12:36.971632 5159 ts_tablet_manager.cc:1909] T d022b5df4a914e439fff918f70c34157 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:36.972538 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 582) took 3594 ms. Trace:
I20240503 02:12:36.972640 5159 rpcz_store.cc:276] 0503 02:12:33.377935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.378545 (+ 610us) service_pool.cc:225] Handling call
0503 02:12:36.935249 (+3556704us) ts_tablet_manager.cc:1120] Deleting tablet d022b5df4a914e439fff918f70c34157
0503 02:12:36.935257 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.946966 (+ 11709us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.971047 (+ 24081us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.972513 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556400}
I20240503 02:12:36.966944 5114 ts_tablet_manager.cc:1888] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.973260 5114 log.cc:1196] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/617ad14927c1441b800a204aefc2e619
I20240503 02:12:36.973652 5114 ts_tablet_manager.cc:1909] T 617ad14927c1441b800a204aefc2e619 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.974537 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 553) took 4173 ms. Trace:
I20240503 02:12:36.974633 5114 rpcz_store.cc:276] 0503 02:12:32.800839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.802879 (+ 2040us) service_pool.cc:225] Handling call
0503 02:12:36.959248 (+4156369us) ts_tablet_manager.cc:1120] Deleting tablet 617ad14927c1441b800a204aefc2e619
0503 02:12:36.959255 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.963653 (+ 4398us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.966833 (+ 3180us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.974510 (+ 7677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4156126}
I20240503 02:12:36.975040 5114 tablet_replica.cc:330] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:36.975272 5114 raft_consensus.cc:2229] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.975458 5114 raft_consensus.cc:2258] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.976133 5114 ts_tablet_manager.cc:1875] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.977099 5115 ts_tablet_manager.cc:1875] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.963717 5103 ts_tablet_manager.cc:1398] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:36.981334 5103 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:36.983265 5103 tablet_bootstrap.cc:654] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.984422 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178760c21904207b6c8317093871f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:36.985742 4667 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 3.
I20240503 02:12:36.986850 4462 leader_election.cc:302] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:36.987293 6183 raft_consensus.cc:2781] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:36.987408 6183 raft_consensus.cc:686] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:36.987651 6183 consensus_queue.cc:237] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:36.988329 5159 tablet_replica.cc:330] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:36.988634 5159 raft_consensus.cc:2229] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:36.988839 5159 raft_consensus.cc:2258] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:36.989614 5159 ts_tablet_manager.cc:1875] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:36.995494 5114 ts_tablet_manager.cc:1888] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:36.995641 5114 log.cc:1196] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/5dbbfa2d1a49400c809d5b476036cdcd
I20240503 02:12:36.996053 5114 ts_tablet_manager.cc:1909] T 5dbbfa2d1a49400c809d5b476036cdcd P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:36.916839 6262 raft_consensus.cc:504] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.001585 6262 leader_election.cc:288] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.001823 4669 raft_consensus.cc:2379] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
W20240503 02:12:37.003028 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 144950us
W20240503 02:12:37.003351 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 138528us
W20240503 02:12:37.003703 4746 leader_election.cc:341] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.003824 4608 leader_election.cc:302] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.004103 6024 raft_consensus.cc:2781] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.004200 6024 raft_consensus.cc:686] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:37.004410 6024 consensus_queue.cc:237] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.005398 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.005596 4806 raft_consensus.cc:2454] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:37.007390 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.007560 4806 raft_consensus.cc:3037] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.010248 4806 raft_consensus.cc:2454] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:37.010679 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885129a4efc5475b97b7ac68e1db5e1d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.010895 4806 raft_consensus.cc:2379] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:37.011222 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14241dd76864e8a93d0279d3ab8f3ac" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.011361 4806 raft_consensus.cc:3037] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:37.013968 4806 raft_consensus.cc:2454] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 3.
I20240503 02:12:37.018132 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:37.018370 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
W20240503 02:12:37.021310 4273 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 85) took 1123 ms (client timeout 1000 ms). Trace:
W20240503 02:12:37.021472 4273 rpcz_store.cc:269] 0503 02:12:35.897715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.982804 (+ 85089us) service_pool.cc:225] Handling call
0503 02:12:36.014623 (+ 31819us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.021151 (+1006528us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":71429888}
I20240503 02:12:37.023955 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:37.016602 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet d022b5df4a914e439fff918f70c34157 (table default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]) successfully deleted
I20240503 02:12:37.024153 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]). Expected version 1 got 0
I20240503 02:12:37.024492 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:37.024848 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
W20240503 02:12:37.025455 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fcfb174825f1441f861ae6816137f855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcfb174825f1441f861ae6816137f855 already in progress: creating tablet
W20240503 02:12:37.025861 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 17a24b5c83b649e79644b08253d851d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a24b5c83b649e79644b08253d851d6 already in progress: creating tablet
W20240503 02:12:37.026095 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 552dad16f30a410280c87a6c5ef0eaa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552dad16f30a410280c87a6c5ef0eaa4 already in progress: creating tablet
W20240503 02:12:37.026337 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 50a43e428b804729adf9d8a3133c90ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50a43e428b804729adf9d8a3133c90ff already in progress: creating tablet
I20240503 02:12:37.026536 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 617ad14927c1441b800a204aefc2e619 (table default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]) successfully deleted
I20240503 02:12:37.025341 4270 catalog_manager.cc:5484] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:37.027262 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7d919b9262ac48cbb8b3f893918b5590 (table default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95]). Expected version 3 got 0
I20240503 02:12:37.027532 4270 catalog_manager.cc:5484] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:37.029141 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 555) took 4225 ms. Trace:
I20240503 02:12:37.029140 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:37.029235 5114 rpcz_store.cc:276] 0503 02:12:32.803413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.803462 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:36.974983 (+4171521us) ts_tablet_manager.cc:1120] Deleting tablet 5dbbfa2d1a49400c809d5b476036cdcd
0503 02:12:36.974995 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.979758 (+ 4763us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:36.995344 (+ 15586us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.029108 (+ 33764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4171293}
I20240503 02:12:37.029596 5114 tablet_replica.cc:330] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.029778 5114 raft_consensus.cc:2229] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:37.030105 5114 raft_consensus.cc:2258] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.030186 4608 leader_election.cc:302] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.030622 4608 leader_election.cc:302] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.030756 5114 ts_tablet_manager.cc:1875] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.031235 4608 leader_election.cc:398] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) with higher term. Message: Invalid argument: T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 for earlier term 2. Current term is 3.
I20240503 02:12:37.031297 4608 leader_election.cc:401] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 02:12:37.038115 5159 ts_tablet_manager.cc:1888] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.038245 5159 log.cc:1196] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3e6a3ca705e8482a8cd6a6fb17d93470
I20240503 02:12:37.038585 5159 ts_tablet_manager.cc:1909] T 3e6a3ca705e8482a8cd6a6fb17d93470 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:37.041553 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65774b69a8bd4664a6b55b84eba37f7f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:37.042026 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cdd9fdae843fe86c779e8eb67bfe2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.042251 4521 raft_consensus.cc:3037] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.043453 6057 raft_consensus.cc:482] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.043612 6057 raft_consensus.cc:504] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:37.044236 6057 leader_election.cc:288] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.044281 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 583) took 3666 ms. Trace:
I20240503 02:12:37.044343 5159 rpcz_store.cc:276] 0503 02:12:33.378028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.378889 (+ 861us) service_pool.cc:225] Handling call
0503 02:12:36.988244 (+3609355us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:36.988263 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.024495 (+ 36232us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.037996 (+ 13501us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.044262 (+ 6266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3609163}
I20240503 02:12:37.044685 5159 tablet_replica.cc:330] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:37.044863 5159 raft_consensus.cc:2229] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.045029 5159 raft_consensus.cc:2258] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.045083 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e9faaa6a584ec38b9da9d2670e47f3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.045521 5159 ts_tablet_manager.cc:1875] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.045908 5105 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:37.046031 5105 ts_tablet_manager.cc:1367] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.163s user 0.010s sys 0.012s
I20240503 02:12:37.046792 5105 raft_consensus.cc:348] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.046988 5105 raft_consensus.cc:374] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.047042 5105 raft_consensus.cc:729] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.047233 5105 consensus_queue.cc:260] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.047641 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cdd9fdae843fe86c779e8eb67bfe2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:37.047729 5105 ts_tablet_manager.cc:1398] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:37.047813 4671 raft_consensus.cc:3037] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.048002 5105 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:37.049091 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e9faaa6a584ec38b9da9d2670e47f3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:37.049301 4672 raft_consensus.cc:3037] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:37.049531 4746 leader_election.cc:341] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.049636 4746 leader_election.cc:302] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.050307 6057 raft_consensus.cc:2726] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.051460 5114 ts_tablet_manager.cc:1888] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.051591 5114 log.cc:1196] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:37.051548 6057 consensus_queue.cc:1035] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.051927 5114 ts_tablet_manager.cc:1909] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.052062 4672 raft_consensus.cc:2454] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:37.053040 6057 consensus_queue.cc:1035] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:37.053870 4746 leader_election.cc:341] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.054004 4746 leader_election.cc:302] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.054859 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65774b69a8bd4664a6b55b84eba37f7f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:37.055217 6057 raft_consensus.cc:2726] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.055591 4746 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:37.056265 6057 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:12:37.056344 6057 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.056416 6057 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:37.056694 4671 raft_consensus.cc:2454] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:37.058353 6247 raft_consensus.cc:2781] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:37.058465 6247 raft_consensus.cc:482] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.058539 6247 raft_consensus.cc:3037] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.058740 4521 raft_consensus.cc:2454] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:37.059507 6057 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.059942 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 554) took 4256 ms. Trace:
I20240503 02:12:37.060026 5114 rpcz_store.cc:276] 0503 02:12:32.803276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.803808 (+ 532us) service_pool.cc:225] Handling call
0503 02:12:37.029540 (+4225732us) ts_tablet_manager.cc:1120] Deleting tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
0503 02:12:37.029554 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.041853 (+ 12299us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.051315 (+ 9462us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.059921 (+ 8606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4225533}
I20240503 02:12:37.060272 6057 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 3 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.061177 6247 raft_consensus.cc:504] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:37.061439 4462 leader_election.cc:341] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3b41dce88919456aa846dbdcf5e2759f
I20240503 02:12:37.061549 4462 leader_election.cc:302] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:37.061772 6247 leader_election.cc:288] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.061918 6247 raft_consensus.cc:2781] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:37.061925 6013 raft_consensus.cc:2726] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:12:37.061980 6247 raft_consensus.cc:482] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.062039 6247 raft_consensus.cc:3037] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:37.062309 4462 consensus_peers.cc:485] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8750b02c5936412db525d2d3e77befeb. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:37.062595 4462 leader_election.cc:341] T 09005fb01c384122a7310c1eea12d3f0 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 09005fb01c384122a7310c1eea12d3f0
I20240503 02:12:37.064424 5159 ts_tablet_manager.cc:1888] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.064538 5159 log.cc:1196] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:37.064422 6247 raft_consensus.cc:504] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.064778 4462 leader_election.cc:302] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:37.064873 5159 ts_tablet_manager.cc:1909] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:37.065060 6013 raft_consensus.cc:2781] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.065052 6247 leader_election.cc:288] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.065173 6013 raft_consensus.cc:686] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:37.065388 6013 consensus_queue.cc:237] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.065532 6247 consensus_queue.cc:1035] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.065917 6247 raft_consensus.cc:2726] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) with higher term. Message: Invalid argument: T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 for earlier term 2. Current term is 3.
I20240503 02:12:37.066675 4462 leader_election.cc:302] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:37.067059 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 061742d380b1496bbfaaa45ead1d0214 (DEFAULT_TABLE table=default.Table_0d42c9ff3ee949f09e9bd50c70d99727 [id=0adf25c7664142c2b16d63f77b360c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.067457 6172 raft_consensus.cc:3037] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.067783 4655 tablet_service.cc:1467] Processing CreateTablet for tablet c5c5b08b5efc46cabfd73aaf73b34088 (DEFAULT_TABLE table=default.Table_2ea009902d854d4096d91c7e2d54a289 [id=88e9b126f30e452fb3ce5dc5ae3650bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.068315 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 (DEFAULT_TABLE table=default.Table_5904b0f9c29f483eab4730665a8bb2fd [id=51ad12aaede24d938edb1f16cf568f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.068845 4648 tablet_service.cc:1467] Processing CreateTablet for tablet f5e305683d70467bacf2c6766e0a61b0 (DEFAULT_TABLE table=default.Table_04087507235042c2831d1475b54a2eb8 [id=524f11367c0c4d0299df57099ed43112]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.069258 5114 tablet_replica.cc:330] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.069449 5114 raft_consensus.cc:2229] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:37.069408 4651 tablet_service.cc:1467] Processing CreateTablet for tablet b94c407d27984998a550bef0d6f48c33 (DEFAULT_TABLE table=default.Table_72c760ead308416095fdc381a2737426 [id=5dc985f1f8e74f13ad768c4b0ad9fa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.069731 5114 raft_consensus.cc:2258] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.069967 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 (DEFAULT_TABLE table=default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.070050 6172 raft_consensus.cc:2726] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.070250 5114 ts_tablet_manager.cc:1875] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.070574 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 13e31219b2134fa080572238ff0be424 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.074506 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.074791 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.074909 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.074942 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075068 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075250 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075412 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075605 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f053537029a44d5a519b612d09d19a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075771 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a14c4f785f4514ae74b8a0b3073503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.075925 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:37.071925 4746 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:37.076211 4746 leader_election.cc:341] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.076786 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 584) took 3698 ms. Trace:
I20240503 02:12:37.076874 6057 consensus_queue.cc:1035] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.077790 4746 leader_election.cc:302] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:37.078344 6057 raft_consensus.cc:2781] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.078469 6057 raft_consensus.cc:686] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:37.070575 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41dce88919456aa846dbdcf5e2759f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.080963 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.224272 6294 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:12:37.224856 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 19fb0929bd234788b5313d96d2e4845c (table default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]) successfully deleted
W20240503 02:12:37.404227 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 179584us
W20240503 02:12:37.404546 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4c0745a55223475893edd01d904a223b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0745a55223475893edd01d904a223b already in progress: creating tablet
W20240503 02:12:37.404798 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 46005e39ab284da6bf1bb9565318a2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46005e39ab284da6bf1bb9565318a2f1 already in progress: creating tablet
I20240503 02:12:37.087538 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:37.097420 5114 ts_tablet_manager.cc:1888] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:37.405031 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 818788a24cd1419980c0d622fb05683b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818788a24cd1419980c0d622fb05683b already in progress: creating tablet
I20240503 02:12:37.405089 5114 log.cc:1196] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/8750b02c5936412db525d2d3e77befeb
I20240503 02:12:37.081794 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178760c21904207b6c8317093871f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:37.405241 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d35138f04f224475af324e77aa1e0191 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35138f04f224475af324e77aa1e0191 already in progress: creating tablet
I20240503 02:12:37.078692 6057 consensus_queue.cc:237] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.089017 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c5b08b5efc46cabfd73aaf73b34088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.405545 5114 ts_tablet_manager.cc:1909] T 8750b02c5936412db525d2d3e77befeb P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.111032 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.406355 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 561) took 4511 ms. Trace:
I20240503 02:12:37.076874 5159 rpcz_store.cc:276] 0503 02:12:33.378149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.379103 (+ 954us) service_pool.cc:225] Handling call
0503 02:12:37.044633 (+3665530us) ts_tablet_manager.cc:1120] Deleting tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
0503 02:12:37.044644 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.060813 (+ 16169us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.064305 (+ 3492us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.076765 (+ 12460us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":3665305}
I20240503 02:12:37.406504 4520 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:37.406777 5159 tablet_replica.cc:330] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:37.406450 5114 rpcz_store.cc:276] 0503 02:12:32.895351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:32.895423 (+ 72us) service_pool.cc:225] Handling call
0503 02:12:37.069205 (+4173782us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:37.069216 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.078509 (+ 9293us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.097034 (+ 18525us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.406333 (+309299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4101894}
I20240503 02:12:37.407289 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 563) took 4371 ms. Trace:
I20240503 02:12:37.407359 5114 rpcz_store.cc:276] 0503 02:12:33.035472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.035690 (+ 218us) service_pool.cc:225] Handling call
0503 02:12:37.407155 (+4371465us) ts_tablet_manager.cc:1120] Deleting tablet 0700d4aed530413983381d1dc4f78ef9
0503 02:12:37.407165 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.407273 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4364633}
I20240503 02:12:37.407660 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 564) took 4367 ms. Trace:
I20240503 02:12:37.407723 5114 rpcz_store.cc:276] 0503 02:12:33.040271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.040581 (+ 310us) service_pool.cc:225] Handling call
0503 02:12:37.407557 (+4366976us) ts_tablet_manager.cc:1120] Deleting tablet 4c0745a55223475893edd01d904a223b
0503 02:12:37.407564 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.407643 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4362924}
I20240503 02:12:37.407987 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 565) took 4208 ms. Trace:
I20240503 02:12:37.408042 5114 rpcz_store.cc:276] 0503 02:12:33.199898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.199977 (+ 79us) service_pool.cc:225] Handling call
0503 02:12:37.407903 (+4207926us) ts_tablet_manager.cc:1120] Deleting tablet 46005e39ab284da6bf1bb9565318a2f1
0503 02:12:37.407910 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.407971 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4207579}
I20240503 02:12:37.408279 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Bootstrap starting.
I20240503 02:12:37.408337 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 566) took 4193 ms. Trace:
I20240503 02:12:37.408401 5114 rpcz_store.cc:276] 0503 02:12:33.214735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.215065 (+ 330us) service_pool.cc:225] Handling call
0503 02:12:37.408249 (+4193184us) ts_tablet_manager.cc:1120] Deleting tablet 818788a24cd1419980c0d622fb05683b
0503 02:12:37.408257 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.408322 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4192954}
I20240503 02:12:37.408653 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 567) took 4130 ms. Trace:
I20240503 02:12:37.408707 5114 rpcz_store.cc:276] 0503 02:12:33.278073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.297480 (+ 19407us) service_pool.cc:225] Handling call
0503 02:12:37.408577 (+4111097us) ts_tablet_manager.cc:1120] Deleting tablet d35138f04f224475af324e77aa1e0191
0503 02:12:37.408585 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.408642 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4110876}
I20240503 02:12:37.089663 5999 raft_consensus.cc:482] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.408960 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 568) took 4094 ms. Trace:
I20240503 02:12:37.409024 5114 rpcz_store.cc:276] 0503 02:12:33.314750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.314808 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:37.408861 (+4094053us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:37.408867 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.408948 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4093823}
I20240503 02:12:37.409025 5999 raft_consensus.cc:504] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.409291 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 569) took 4090 ms. Trace:
I20240503 02:12:37.409366 5114 rpcz_store.cc:276] 0503 02:12:33.319281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.319416 (+ 135us) service_pool.cc:225] Handling call
0503 02:12:37.409202 (+4089786us) ts_tablet_manager.cc:1120] Deleting tablet 76aef6b021da4f5aad7f523b15d484ec
0503 02:12:37.409210 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.409276 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4089518}
I20240503 02:12:37.409683 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 570) took 4067 ms. Trace:
I20240503 02:12:37.409688 5999 leader_election.cc:288] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.409756 5114 rpcz_store.cc:276] 0503 02:12:33.342327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.342384 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:37.409590 (+4067206us) ts_tablet_manager.cc:1120] Deleting tablet c757a3a979c2489daa621cf8ea293ea8
0503 02:12:37.409598 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.409668 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4066959}
I20240503 02:12:37.409863 4520 raft_consensus.cc:2454] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 3.
W20240503 02:12:37.114324 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 47610.9us
I20240503 02:12:37.410017 5114 tablet_replica.cc:330] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.410282 5114 raft_consensus.cc:2229] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.144109 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 (DEFAULT_TABLE table=default.Table_950e46a8a847487e902ad641c4895f8c [id=a9623c75693d4b319c97d49e3d4a368e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.410605 5159 raft_consensus.cc:2229] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.410774 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7219ce14a9ae4488b97914b9d02995f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.410872 5159 raft_consensus.cc:2258] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.411643 5159 ts_tablet_manager.cc:1875] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.413045 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73198ec769a9486da3f85e8fcbef8e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.099551 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:37.108886 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88eaba979394b449d96ee956c3f28a8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:37.145612 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0303f1b1194115a7142a00470457c3 (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.414134 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0303f1b1194115a7142a00470457c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:37.415444 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8195767d80384a6c9a4690292f6776de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8195767d80384a6c9a4690292f6776de already in progress: creating tablet
W20240503 02:12:37.415903 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 76aef6b021da4f5aad7f523b15d484ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76aef6b021da4f5aad7f523b15d484ec already in progress: creating tablet
W20240503 02:12:37.416141 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c757a3a979c2489daa621cf8ea293ea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c757a3a979c2489daa621cf8ea293ea8 already in progress: creating tablet
I20240503 02:12:37.416329 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 9327a9e5472b4eda854320c3a95f8e56 (table default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]) successfully deleted
I20240503 02:12:37.416738 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (table default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]) successfully deleted
I20240503 02:12:37.416898 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 5dbbfa2d1a49400c809d5b476036cdcd (table default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]) successfully deleted
I20240503 02:12:37.417097 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (table default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]) successfully deleted
I20240503 02:12:37.158862 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89c4be807400eb9f1b125a55e5d27 (DEFAULT_TABLE table=default.Table_75ace3349fd14ae992051f80c21e3896 [id=30e045b34cbc44b8bbebc124702b2b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.164810 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:37.417249 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 8750b02c5936412db525d2d3e77befeb (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]) successfully deleted
I20240503 02:12:37.144322 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.417472 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89c4be807400eb9f1b125a55e5d27. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:37.417462 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0700d4aed530413983381d1dc4f78ef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700d4aed530413983381d1dc4f78ef9 already in progress: creating tablet
W20240503 02:12:37.417677 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4c0745a55223475893edd01d904a223b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0745a55223475893edd01d904a223b already in progress: creating tablet
W20240503 02:12:37.417888 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 46005e39ab284da6bf1bb9565318a2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46005e39ab284da6bf1bb9565318a2f1 already in progress: creating tablet
W20240503 02:12:37.418071 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 818788a24cd1419980c0d622fb05683b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818788a24cd1419980c0d622fb05683b already in progress: creating tablet
W20240503 02:12:37.418267 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d35138f04f224475af324e77aa1e0191 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35138f04f224475af324e77aa1e0191 already in progress: creating tablet
W20240503 02:12:37.418471 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8195767d80384a6c9a4690292f6776de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8195767d80384a6c9a4690292f6776de already in progress: creating tablet
I20240503 02:12:37.418514 5114 raft_consensus.cc:2258] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:37.418679 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 76aef6b021da4f5aad7f523b15d484ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76aef6b021da4f5aad7f523b15d484ec already in progress: creating tablet
I20240503 02:12:37.145360 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08e7d17f1644749a9a2e7175b3191a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.143353 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 (DEFAULT_TABLE table=default.Table_950e46a8a847487e902ad641c4895f8c [id=a9623c75693d4b319c97d49e3d4a368e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:37.418870 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c757a3a979c2489daa621cf8ea293ea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c757a3a979c2489daa621cf8ea293ea8 already in progress: creating tablet
I20240503 02:12:37.419103 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7219ce14a9ae4488b97914b9d02995f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.419174 5114 ts_tablet_manager.cc:1875] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.164907 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 478ac0f515b34e829c9a776bc32d875f (DEFAULT_TABLE table=default.Table_9801d79108b449c69664c004a6c352a9 [id=f4b07d6766254a9b83c09b7339ed6370]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.145851 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:37.420918 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478ac0f515b34e829c9a776bc32d875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.143604 4502 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.172225 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 30d115c3aabf484e88a3d55377038182 (DEFAULT_TABLE table=default.Table_2288fa2ae0674a20b78e456480dd514b [id=af5eeecd5885418394623d9a8c55bd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.421403 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d115c3aabf484e88a3d55377038182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.146477 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0303f1b1194115a7142a00470457c3 (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.424229 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0303f1b1194115a7142a00470457c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:37.424540 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8debae3d66514458b2f2ac04982975b8 consensus state has no leader
I20240503 02:12:37.424742 4275 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 870) took 1485 ms. Trace:
I20240503 02:12:37.424834 4275 rpcz_store.cc:276] 0503 02:12:35.939049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.854603 (+915554us) service_pool.cc:225] Handling call
0503 02:12:36.854919 (+ 316us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:36.855112 (+ 193us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:36.855297 (+ 185us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:36.855379 (+ 82us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:37.424272 (+568893us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:37.424680 (+ 408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.900281 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.900426 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.900434 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:36.900437 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:36.900576 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.900590 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.900595 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:36.900679 (+ 84us) tablet.cc:699] Row locks acquired
0503 02:12:36.900682 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.900725 (+ 43us) write_op.cc:270] Start()
0503 02:12:36.900763 (+ 38us) write_op.cc:276] Timestamp: P: 1714702356900663 usec, L: 0
0503 02:12:36.900766 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.901198 (+ 432us) log.cc:842] Serialized 405 byte log entry
0503 02:12:37.417202 (+516004us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.417670 (+ 468us) write_op.cc:301] APPLY: starting
0503 02:12:37.417692 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.417735 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.417791 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.417853 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.417899 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:37.417907 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.417998 (+ 91us) log.cc:842] Serialized 23 byte log entry
0503 02:12:37.418076 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.418097 (+ 21us) write_op.cc:454] Released schema lock
0503 02:12:37.418105 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.418174 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.418178 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":394,"apply.run_cpu_time_us":571,"apply.run_wall_time_us":572,"num_ops":1,"prepare.queue_time_us":44378,"prepare.run_cpu_time_us":907,"prepare.run_wall_time_us":1049,"raft.queue_time_us":889,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":516401}]]}
I20240503 02:12:37.427023 4272 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 871) took 1487 ms. Trace:
I20240503 02:12:37.427162 4272 rpcz_store.cc:276] 0503 02:12:35.939234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.858508 (+919274us) service_pool.cc:225] Handling call
0503 02:12:36.859390 (+ 882us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:36.859871 (+ 481us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.426749 (+566878us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.426887 (+ 138us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.426980 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.901347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.901466 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.901473 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.901475 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:36.901558 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.901568 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.901572 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.901828 (+ 256us) tablet.cc:699] Row locks acquired
0503 02:12:36.901831 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.901864 (+ 33us) write_op.cc:270] Start()
0503 02:12:36.901894 (+ 30us) write_op.cc:276] Timestamp: P: 1714702356901806 usec, L: 0
0503 02:12:36.901897 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.902047 (+ 150us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.417286 (+515239us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.418267 (+ 981us) write_op.cc:301] APPLY: starting
0503 02:12:37.418278 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.418314 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.418361 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.418466 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.418496 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.418503 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:37.418555 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.418594 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.418608 (+ 14us) write_op.cc:454] Released schema lock
0503 02:12:37.418613 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.418657 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.418661 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":947,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":437,"num_ops":2,"prepare.queue_time_us":41019,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":818,"raft.queue_time_us":92,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":515373,"spinlock_wait_cycles":54656,"threads_started":1}]]}
I20240503 02:12:37.144850 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89c4be807400eb9f1b125a55e5d27 (DEFAULT_TABLE table=default.Table_75ace3349fd14ae992051f80c21e3896 [id=30e045b34cbc44b8bbebc124702b2b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.423094 5159 ts_tablet_manager.cc:1888] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.429076 5159 log.cc:1196] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:37.172881 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce41298154225be9599981eb31ff7 (DEFAULT_TABLE table=default.Table_3fe44eb302634e1a8900af7fba21d5b2 [id=20c67f0b83d94df4834a90bea68f1041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.147009 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 (DEFAULT_TABLE table=default.Table_950e46a8a847487e902ad641c4895f8c [id=a9623c75693d4b319c97d49e3d4a368e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.429517 5159 ts_tablet_manager.cc:1909] T e94031465ce7407dab0f90ad8809ac21 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:37.429621 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7219ce14a9ae4488b97914b9d02995f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.430877 5114 ts_tablet_manager.cc:1888] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.430984 5114 log.cc:1196] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:37.431372 5114 ts_tablet_manager.cc:1909] T 36a14c4f785f4514ae74b8a0b3073503 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.159482 4504 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.173209 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 6344396cb30a4194b35f9f297592f93b (DEFAULT_TABLE table=default.Table_68c5b84eae2d40739236f3f03dc38fdf [id=99196ca7ab0c41e5b71766eee52803fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.196316 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:37.163997 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 478ac0f515b34e829c9a776bc32d875f (DEFAULT_TABLE table=default.Table_9801d79108b449c69664c004a6c352a9 [id=f4b07d6766254a9b83c09b7339ed6370]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.210263 5106 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:37.165645 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 30d115c3aabf484e88a3d55377038182 (DEFAULT_TABLE table=default.Table_2288fa2ae0674a20b78e456480dd514b [id=af5eeecd5885418394623d9a8c55bd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.172699 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce41298154225be9599981eb31ff7 (DEFAULT_TABLE table=default.Table_3fe44eb302634e1a8900af7fba21d5b2 [id=20c67f0b83d94df4834a90bea68f1041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.434092 5106 ts_tablet_manager.cc:1367] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.515s user 0.005s sys 0.018s
I20240503 02:12:37.173696 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef (DEFAULT_TABLE table=default.Table_a6a57c6d925744ca8d5983412f5f516f [id=5b25f6cab1254b09807146e48508cc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.174628 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8809600c24d339e556ac4edbe4e83 (DEFAULT_TABLE table=default.Table_9f8e5dd83dd347d29e90e271ed77b18a [id=0d8b79097a7d4fd3a4d6bfd4062f5527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.173398 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 6344396cb30a4194b35f9f297592f93b (DEFAULT_TABLE table=default.Table_68c5b84eae2d40739236f3f03dc38fdf [id=99196ca7ab0c41e5b71766eee52803fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.194018 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.173952 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef (DEFAULT_TABLE table=default.Table_a6a57c6d925744ca8d5983412f5f516f [id=5b25f6cab1254b09807146e48508cc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.434430 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061742d380b1496bbfaaa45ead1d0214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.434434 4806 raft_consensus.cc:3037] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.174592 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8809600c24d339e556ac4edbe4e83 (DEFAULT_TABLE table=default.Table_9f8e5dd83dd347d29e90e271ed77b18a [id=0d8b79097a7d4fd3a4d6bfd4062f5527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.175233 4501 tablet_service.cc:1467] Processing CreateTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 (DEFAULT_TABLE table=default.Table_fb49809c3ca04cf9ac1cad7770bd1897 [id=6ab31ee3923149daa4472d116ba739af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.202298 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 1716c4807e004760b34c0492cb7f522d (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.205951 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 (DEFAULT_TABLE table=default.Table_8fa27e930c344572afb5f6ba668633df [id=b077f99296dc4b6f97e8b1808ca7968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.208632 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e80bdc70d45e8ae0c7bb162654868 (DEFAULT_TABLE table=default.Table_73be651ef93740b19d0d19abe2aff424 [id=c9e4b848de544329ab3f83083ba4f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.434857 5106 raft_consensus.cc:348] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.435107 5106 raft_consensus.cc:374] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.212829 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:37.435176 5106 raft_consensus.cc:729] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.435382 5106 consensus_queue.cc:260] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.435865 5106 ts_tablet_manager.cc:1398] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:37.436095 5106 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:37.224849 5115 ts_tablet_manager.cc:1888] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.437001 5115 log.cc:1196] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b080fc5ffc564420a0b7f2b7460ebbbb
I20240503 02:12:37.437052 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e31219b2134fa080572238ff0be424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.437320 5115 ts_tablet_manager.cc:1909] T b080fc5ffc564420a0b7f2b7460ebbbb P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:37.437583 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 571) took 4066 ms. Trace:
I20240503 02:12:37.437636 5115 rpcz_store.cc:276] 0503 02:12:33.371193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.371596 (+ 403us) service_pool.cc:225] Handling call
0503 02:12:36.946252 (+3574656us) ts_tablet_manager.cc:1120] Deleting tablet b080fc5ffc564420a0b7f2b7460ebbbb
0503 02:12:36.946261 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:36.980582 (+ 34321us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.224725 (+244143us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.437565 (+212840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3574462}
I20240503 02:12:37.437862 5115 tablet_replica.cc:330] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:37.437978 5115 raft_consensus.cc:2229] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.438076 5115 raft_consensus.cc:2258] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.438481 5115 ts_tablet_manager.cc:1875] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.196487 5105 tablet_bootstrap.cc:654] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:37.233734 4608 consensus_peers.cc:485] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b41dce88919456aa846dbdcf5e2759f. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:37.440475 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 206775us
I20240503 02:12:37.440722 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e305683d70467bacf2c6766e0a61b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.201006 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 (DEFAULT_TABLE table=default.Table_fb49809c3ca04cf9ac1cad7770bd1897 [id=6ab31ee3923149daa4472d116ba739af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.204778 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 1716c4807e004760b34c0492cb7f522d (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.214087 4490 tablet_service.cc:1467] Processing CreateTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 (DEFAULT_TABLE table=default.Table_cab73492219d41c4bf3fafcf6ac31309 [id=16de9c3b221b4c29a8d9452403f8a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.207509 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 (DEFAULT_TABLE table=default.Table_8fa27e930c344572afb5f6ba668633df [id=b077f99296dc4b6f97e8b1808ca7968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.215528 4500 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2c4c285c14857a7403a74d55f11ee (DEFAULT_TABLE table=default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f [id=4dfd792d0fc14d4ca6c7b16f4e4a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.211591 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e80bdc70d45e8ae0c7bb162654868 (DEFAULT_TABLE table=default.Table_73be651ef93740b19d0d19abe2aff424 [id=c9e4b848de544329ab3f83083ba4f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.218394 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 (DEFAULT_TABLE table=default.Table_8f8f3c9fd776473b8dd8eed18d12c897 [id=45f956d540154b739ebce781dba884a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.232875 6247 raft_consensus.cc:482] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.215166 4791 tablet_service.cc:1467] Processing CreateTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 (DEFAULT_TABLE table=default.Table_cab73492219d41c4bf3fafcf6ac31309 [id=16de9c3b221b4c29a8d9452403f8a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.224645 5103 tablet_bootstrap.cc:492] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:37.441480 5103 ts_tablet_manager.cc:1367] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.460s user 0.034s sys 0.081s
I20240503 02:12:37.441566 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94c407d27984998a550bef0d6f48c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.442282 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f3232d509f4cb39eb8923fe1d120a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.442283 5103 raft_consensus.cc:348] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.442477 5103 raft_consensus.cc:374] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.442534 5103 raft_consensus.cc:729] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.442723 5103 consensus_queue.cc:260] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.443230 5103 ts_tablet_manager.cc:1398] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:37.217448 4783 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2c4c285c14857a7403a74d55f11ee (DEFAULT_TABLE table=default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f [id=4dfd792d0fc14d4ca6c7b16f4e4a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.402595 6013 raft_consensus.cc:482] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.443462 6013 raft_consensus.cc:504] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.444133 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89c4be807400eb9f1b125a55e5d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.220116 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 (DEFAULT_TABLE table=default.Table_8f8f3c9fd776473b8dd8eed18d12c897 [id=45f956d540154b739ebce781dba884a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.444823 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb59bbf7b68431c86d88f27ef2528b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.229915 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.445683 4805 raft_consensus.cc:3037] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.230139 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.445971 4813 raft_consensus.cc:3037] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.447199 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3ce41298154225be9599981eb31ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.447669 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344396cb30a4194b35f9f297592f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.230403 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.449080 4808 raft_consensus.cc:2379] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:37.126075 4746 leader_election.cc:302] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:37.449590 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 323566us
I20240503 02:12:37.450183 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2c4c285c14857a7403a74d55f11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.450408 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.450621 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb59bbf7b68431c86d88f27ef2528b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.453208 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.455149 6317 consensus_queue.cc:1035] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.456668 4268 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 867) took 1663 ms. Trace:
I20240503 02:12:37.456790 4268 rpcz_store.cc:276] 0503 02:12:35.793394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.738951 (+945557us) service_pool.cc:225] Handling call
0503 02:12:36.744974 (+ 6023us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:36.942588 (+197614us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:36.943030 (+ 442us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.456278 (+513248us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.456458 (+ 180us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.456585 (+ 127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.943708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.943809 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.943816 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.943818 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:36.943906 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.943918 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.943923 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.943998 (+ 75us) tablet.cc:699] Row locks acquired
0503 02:12:36.944001 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.944037 (+ 36us) write_op.cc:270] Start()
0503 02:12:36.944071 (+ 34us) write_op.cc:276] Timestamp: P: 1714702356944019 usec, L: 0
0503 02:12:36.944075 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.944266 (+ 191us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.453064 (+508798us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.453468 (+ 404us) write_op.cc:301] APPLY: starting
0503 02:12:37.453488 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.453534 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.453565 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.453692 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.453731 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.453739 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.453825 (+ 86us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.453915 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.453939 (+ 24us) write_op.cc:454] Released schema lock
0503 02:12:37.453946 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.454001 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.454005 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":12103552,"child_traces":[["op",{"apply.queue_time_us":336,"apply.run_cpu_time_us":594,"apply.run_wall_time_us":595,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":676,"raft.queue_time_us":3405,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":508960}]]}
I20240503 02:12:37.459106 4267 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 877) took 1272 ms. Trace:
I20240503 02:12:37.459209 4267 rpcz_store.cc:276] 0503 02:12:36.186188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.945121 (+758933us) service_pool.cc:225] Handling call
0503 02:12:36.946043 (+ 922us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:36.946508 (+ 465us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.458838 (+512330us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.458954 (+ 116us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.459048 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.947073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.947173 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.947181 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:36.947183 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:36.947268 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.947280 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.947284 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.947360 (+ 76us) tablet.cc:699] Row locks acquired
0503 02:12:36.947363 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.947400 (+ 37us) write_op.cc:270] Start()
0503 02:12:36.947437 (+ 37us) write_op.cc:276] Timestamp: P: 1714702356947380 usec, L: 0
0503 02:12:36.947450 (+ 13us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.947646 (+ 196us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.453176 (+505530us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.454089 (+ 913us) write_op.cc:301] APPLY: starting
0503 02:12:37.454100 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.454148 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.454180 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.454294 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.454323 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.454330 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:37.454383 (+ 53us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.454452 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.454467 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:37.454472 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.454515 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.454518 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":881,"apply.run_cpu_time_us":468,"apply.run_wall_time_us":467,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":681,"prepare.run_wall_time_us":681,"raft.queue_time_us":98,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":505685}]]}
W20240503 02:12:37.460320 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54959b3e30944fab8bd5fd47ce42d246 consensus state has no leader
I20240503 02:12:37.460494 4274 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 866) took 1667 ms. Trace:
I20240503 02:12:37.460592 4274 rpcz_store.cc:276] 0503 02:12:35.793247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.738363 (+945116us) service_pool.cc:225] Handling call
0503 02:12:36.744592 (+ 6229us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:36.952844 (+208252us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:36.953221 (+ 377us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:36.953429 (+ 208us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:36.953533 (+ 104us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:37.460074 (+506541us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:37.460463 (+ 389us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.954227 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.954324 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.954331 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.954334 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:36.954431 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.954451 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.954456 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:36.954520 (+ 64us) tablet.cc:699] Row locks acquired
0503 02:12:36.954523 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.954561 (+ 38us) write_op.cc:270] Start()
0503 02:12:36.954606 (+ 45us) write_op.cc:276] Timestamp: P: 1714702356954541 usec, L: 0
0503 02:12:36.954609 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.954797 (+ 188us) log.cc:842] Serialized 405 byte log entry
0503 02:12:37.453228 (+498431us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.454583 (+ 1355us) write_op.cc:301] APPLY: starting
0503 02:12:37.454605 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.454631 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.454651 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.454692 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.454714 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:37.454722 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.454764 (+ 42us) log.cc:842] Serialized 23 byte log entry
0503 02:12:37.454799 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.454809 (+ 10us) write_op.cc:454] Released schema lock
0503 02:12:37.454814 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.454854 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.454858 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":13291904,"child_traces":[["op",{"apply.queue_time_us":1327,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":309,"num_ops":1,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":669,"prepare.run_wall_time_us":669,"raft.queue_time_us":327,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":498603,"threads_started":1}]]}
I20240503 02:12:37.462584 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716c4807e004760b34c0492cb7f522d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.462739 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 585) took 4084 ms. Trace:
I20240503 02:12:37.462880 5159 rpcz_store.cc:276] 0503 02:12:33.378245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.379369 (+ 1124us) service_pool.cc:225] Handling call
0503 02:12:37.406713 (+4027344us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:37.406726 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.415873 (+ 9147us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.422974 (+ 7101us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.462676 (+ 39702us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3530,"tablet-delete.queue_time_us":4027081}
I20240503 02:12:37.463337 5159 tablet_replica.cc:330] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:37.463608 5159 raft_consensus.cc:2229] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.463820 5159 raft_consensus.cc:2258] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.464712 5159 ts_tablet_manager.cc:1875] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.465204 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82190dfb1b1418b9f20bf4d0c2c2409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.469493 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e80bdc70d45e8ae0c7bb162654868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.475955 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8809600c24d339e556ac4edbe4e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.477355 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3ce41298154225be9599981eb31ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.477604 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478ac0f515b34e829c9a776bc32d875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.477790 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82190dfb1b1418b9f20bf4d0c2c2409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.477990 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.478312 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e80bdc70d45e8ae0c7bb162654868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.478485 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8809600c24d339e556ac4edbe4e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.478844 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716c4807e004760b34c0492cb7f522d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.479010 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344396cb30a4194b35f9f297592f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.479502 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 571) took 4107 ms. Trace:
I20240503 02:12:37.479636 5114 rpcz_store.cc:276] 0503 02:12:33.372514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.374226 (+ 1712us) service_pool.cc:225] Handling call
0503 02:12:37.409976 (+4035750us) ts_tablet_manager.cc:1120] Deleting tablet 36a14c4f785f4514ae74b8a0b3073503
0503 02:12:37.409984 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.425240 (+ 15256us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.430771 (+ 5531us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.479473 (+ 48702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4035501}
I20240503 02:12:37.480379 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 36a14c4f785f4514ae74b8a0b3073503 (table default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]) successfully deleted
I20240503 02:12:37.481166 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ba631d85854b43bcb4bece9bfba7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.484308 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.484910 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ba631d85854b43bcb4bece9bfba7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.441320 6247 raft_consensus.cc:504] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.490669 6247 leader_election.cc:288] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.493674 4806 raft_consensus.cc:2454] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:37.497181 5114 tablet_replica.cc:330] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:37.410360 4462 consensus_peers.cc:485] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f053537029a44d5a519b612d09d19a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:37.141572 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0303f1b1194115a7142a00470457c3 (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:37.501091 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 90760.2us
I20240503 02:12:37.501361 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0303f1b1194115a7142a00470457c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.502626 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.503477 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b72c20a4b5a4d2ea266f031733d1aab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.504398 6172 raft_consensus.cc:482] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.504530 6172 raft_consensus.cc:504] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.505123 6172 leader_election.cc:288] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.505335 4525 raft_consensus.cc:1226] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Rejecting Update request from peer 1f50719d94564844b15e1a9211d48a81 for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:37.142639 4509 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.497504 5114 raft_consensus.cc:2229] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.505689 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.505844 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:37.506023 5114 raft_consensus.cc:2258] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.506218 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:37.506444 4667 raft_consensus.cc:2379] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:37.506677 5114 ts_tablet_manager.cc:1875] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.511718 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet e94031465ce7407dab0f90ad8809ac21 (table default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]) successfully deleted
I20240503 02:12:37.444169 6013 leader_election.cc:288] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:37.516429 6296 raft_consensus.cc:2781] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:37.518828 4608 consensus_peers.cc:485] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d022b5df4a914e439fff918f70c34157. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:37.519394 4608 leader_election.cc:302] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:37.520098 6262 consensus_queue.cc:1035] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.520277 6024 raft_consensus.cc:2781] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.520366 6024 raft_consensus.cc:686] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:37.521057 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce41298154225be9599981eb31ff7 (DEFAULT_TABLE table=default.Table_3fe44eb302634e1a8900af7fba21d5b2 [id=20c67f0b83d94df4834a90bea68f1041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.521529 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3ce41298154225be9599981eb31ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.522125 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 30d115c3aabf484e88a3d55377038182 (DEFAULT_TABLE table=default.Table_2288fa2ae0674a20b78e456480dd514b [id=af5eeecd5885418394623d9a8c55bd03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.522509 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d115c3aabf484e88a3d55377038182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.538416 5159 ts_tablet_manager.cc:1888] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.539286 5159 log.cc:1196] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/19fb0929bd234788b5313d96d2e4845c
I20240503 02:12:37.540129 5159 ts_tablet_manager.cc:1909] T 19fb0929bd234788b5313d96d2e4845c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:37.540971 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 586) took 4162 ms. Trace:
I20240503 02:12:37.541602 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d115c3aabf484e88a3d55377038182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.541568 5159 rpcz_store.cc:276] 0503 02:12:33.378344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.379662 (+ 1318us) service_pool.cc:225] Handling call
0503 02:12:37.463251 (+4083589us) ts_tablet_manager.cc:1120] Deleting tablet 19fb0929bd234788b5313d96d2e4845c
0503 02:12:37.463264 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.475398 (+ 12134us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.538248 (+ 62850us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.540893 (+ 2645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4083382}
I20240503 02:12:37.542327 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 19fb0929bd234788b5313d96d2e4845c (table default.Table_5ac4a60d6df343668e711966d75f932c [id=ec3249d83d2044acb117dabb54b21b7c]) successfully deleted
I20240503 02:12:37.542479 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2c4c285c14857a7403a74d55f11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.546644 4813 raft_consensus.cc:2454] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:37.546752 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 (DEFAULT_TABLE table=default.Table_fb49809c3ca04cf9ac1cad7770bd1897 [id=6ab31ee3923149daa4472d116ba739af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.546932 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 478ac0f515b34e829c9a776bc32d875f (DEFAULT_TABLE table=default.Table_9801d79108b449c69664c004a6c352a9 [id=f4b07d6766254a9b83c09b7339ed6370]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.547230 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8809600c24d339e556ac4edbe4e83 (DEFAULT_TABLE table=default.Table_9f8e5dd83dd347d29e90e271ed77b18a [id=0d8b79097a7d4fd3a4d6bfd4062f5527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.549815 5103 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:37.550524 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478ac0f515b34e829c9a776bc32d875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.552054 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89c4be807400eb9f1b125a55e5d27 (DEFAULT_TABLE table=default.Table_75ace3349fd14ae992051f80c21e3896 [id=30e045b34cbc44b8bbebc124702b2b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.552481 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89c4be807400eb9f1b125a55e5d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.553246 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 6344396cb30a4194b35f9f297592f93b (DEFAULT_TABLE table=default.Table_68c5b84eae2d40739236f3f03dc38fdf [id=99196ca7ab0c41e5b71766eee52803fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.553613 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344396cb30a4194b35f9f297592f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.546074 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:37.553941 4667 raft_consensus.cc:2454] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:37.556756 5159 tablet_replica.cc:330] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:37.558111 5159 raft_consensus.cc:2229] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:37.559648 5159 raft_consensus.cc:2258] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.560745 4462 leader_election.cc:302] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:37.565317 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 1716c4807e004760b34c0492cb7f522d (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.565941 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 (DEFAULT_TABLE table=default.Table_8fa27e930c344572afb5f6ba668633df [id=b077f99296dc4b6f97e8b1808ca7968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.561676 6296 raft_consensus.cc:686] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:37.567445 6296 consensus_queue.cc:237] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.572039 5159 ts_tablet_manager.cc:1875] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.561920 4805 raft_consensus.cc:2454] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:37.582921 6183 raft_consensus.cc:482] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.583091 6183 raft_consensus.cc:504] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.586290 5103 tablet_bootstrap.cc:654] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:37.586522 5114 ts_tablet_manager.cc:1888] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:37.584158 6183 leader_election.cc:288] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.586751 5114 log.cc:1196] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/4ced4dfb2cf541af98d50d22d1a0f0ac
I20240503 02:12:37.587292 5114 ts_tablet_manager.cc:1909] T 4ced4dfb2cf541af98d50d22d1a0f0ac P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.587781 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 572) took 4215 ms. Trace:
I20240503 02:12:37.589519 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8809600c24d339e556ac4edbe4e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.596426 5106 tablet_bootstrap.cc:654] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:37.596973 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716c4807e004760b34c0492cb7f522d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.597855 4637 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2c4c285c14857a7403a74d55f11ee (DEFAULT_TABLE table=default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f [id=4dfd792d0fc14d4ca6c7b16f4e4a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.598215 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2c4c285c14857a7403a74d55f11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.520576 6024 consensus_queue.cc:237] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.588069 6013 raft_consensus.cc:2781] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:37.599243 6013 raft_consensus.cc:482] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.599326 6013 raft_consensus.cc:3037] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.587898 5114 rpcz_store.cc:276] 0503 02:12:33.372689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.374519 (+ 1830us) service_pool.cc:225] Handling call
0503 02:12:37.497098 (+4122579us) ts_tablet_manager.cc:1120] Deleting tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
0503 02:12:37.497112 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.523114 (+ 26002us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.586416 (+ 63302us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.587754 (+ 1338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4122399}
I20240503 02:12:37.602911 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 578) took 4176 ms. Trace:
I20240503 02:12:37.603014 5114 rpcz_store.cc:276] 0503 02:12:33.426861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.426917 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:37.602769 (+4175852us) ts_tablet_manager.cc:1120] Deleting tablet 857d019b8ab94be1b7719fec83e57e55
0503 02:12:37.602781 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.602890 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4174723}
I20240503 02:12:37.603297 5114 tablet_replica.cc:330] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.603475 5114 raft_consensus.cc:2229] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:37.603421 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (table default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]) successfully deleted
W20240503 02:12:37.603838 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 857d019b8ab94be1b7719fec83e57e55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857d019b8ab94be1b7719fec83e57e55 already in progress: creating tablet
I20240503 02:12:37.605682 5114 raft_consensus.cc:2258] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.606561 5114 ts_tablet_manager.cc:1875] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.606612 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e80bdc70d45e8ae0c7bb162654868 (DEFAULT_TABLE table=default.Table_73be651ef93740b19d0d19abe2aff424 [id=c9e4b848de544329ab3f83083ba4f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.607117 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 (DEFAULT_TABLE table=default.Table_8f8f3c9fd776473b8dd8eed18d12c897 [id=45f956d540154b739ebce781dba884a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.609341 4651 tablet_service.cc:1467] Processing CreateTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 (DEFAULT_TABLE table=default.Table_cab73492219d41c4bf3fafcf6ac31309 [id=16de9c3b221b4c29a8d9452403f8a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.612748 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef (DEFAULT_TABLE table=default.Table_a6a57c6d925744ca8d5983412f5f516f [id=5b25f6cab1254b09807146e48508cc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.625430 5115 ts_tablet_manager.cc:1888] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:37.625584 5115 log.cc:1196] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/09005fb01c384122a7310c1eea12d3f0
I20240503 02:12:37.625916 5115 ts_tablet_manager.cc:1909] T 09005fb01c384122a7310c1eea12d3f0 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:37.626262 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 576) took 4203 ms. Trace:
I20240503 02:12:37.626353 5115 rpcz_store.cc:276] 0503 02:12:33.422833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.423166 (+ 333us) service_pool.cc:225] Handling call
0503 02:12:37.437832 (+4014666us) ts_tablet_manager.cc:1120] Deleting tablet 09005fb01c384122a7310c1eea12d3f0
0503 02:12:37.437837 (+ 5us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.443923 (+ 6086us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.625307 (+181384us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.626239 (+ 932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4014420}
I20240503 02:12:37.635223 6013 raft_consensus.cc:504] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.636013 6013 leader_election.cc:288] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.639950 5114 ts_tablet_manager.cc:1888] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.640162 5114 log.cc:1196] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/159ea1eb32394b04b72258d24196202b
I20240503 02:12:37.641206 5114 ts_tablet_manager.cc:1909] T 159ea1eb32394b04b72258d24196202b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.641605 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 577) took 4219 ms. Trace:
I20240503 02:12:37.641765 5114 rpcz_store.cc:276] 0503 02:12:33.422589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.426271 (+ 3682us) service_pool.cc:225] Handling call
0503 02:12:37.603253 (+4176982us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:37.603262 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.636604 (+ 33342us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.639846 (+ 3242us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.641583 (+ 1737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4175112}
I20240503 02:12:37.642349 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 580) took 4131 ms. Trace:
I20240503 02:12:37.642500 5114 rpcz_store.cc:276] 0503 02:12:33.510536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.546689 (+ 36153us) service_pool.cc:225] Handling call
0503 02:12:37.642205 (+4095516us) ts_tablet_manager.cc:1120] Deleting tablet 16fcfe6e2dc844a486476bf4be80d5cb
0503 02:12:37.642217 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.642331 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4095256}
I20240503 02:12:37.642944 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 583) took 4045 ms. Trace:
I20240503 02:12:37.643085 5114 rpcz_store.cc:276] 0503 02:12:33.597120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.597181 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:37.642829 (+4045648us) ts_tablet_manager.cc:1120] Deleting tablet dee34be89bda450296b916163bb10c46
0503 02:12:37.642841 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.642927 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4045385}
I20240503 02:12:37.643447 5114 tablet_replica.cc:330] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.645395 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.645511 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 159ea1eb32394b04b72258d24196202b (table default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]) successfully deleted
I20240503 02:12:37.645681 4813 raft_consensus.cc:2454] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:37.645864 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 16fcfe6e2dc844a486476bf4be80d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fcfe6e2dc844a486476bf4be80d5cb already in progress: creating tablet
W20240503 02:12:37.646111 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet dee34be89bda450296b916163bb10c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee34be89bda450296b916163bb10c46 already in progress: creating tablet
I20240503 02:12:37.646402 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b080fc5ffc564420a0b7f2b7460ebbbb (table default.Table_fd0e6dfdbbc14c7599538eb05a30083c [id=e08a62f9895e4721b5524313acf78175]) successfully deleted
I20240503 02:12:37.648273 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.648531 4806 raft_consensus.cc:2454] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:37.653259 4805 raft_consensus.cc:1226] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Rejecting Update request from peer 1f50719d94564844b15e1a9211d48a81 for earlier term 2. Current term is 3. Ops: []
I20240503 02:12:37.653477 4528 raft_consensus.cc:1226] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Rejecting Update request from peer 1f50719d94564844b15e1a9211d48a81 for earlier term 2. Current term is 3. Ops: []
I20240503 02:12:37.662163 6247 consensus_queue.cc:1035] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:37.663568 4608 leader_election.cc:341] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.666754 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
W20240503 02:12:37.667168 4608 consensus_peers.cc:485] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:37.669800 5114 raft_consensus.cc:2229] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.672304 4608 leader_election.cc:302] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.683301 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:37.683612 4521 raft_consensus.cc:2379] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:37.683179 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:37.684391 5114 raft_consensus.cc:2258] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.691601 5114 ts_tablet_manager.cc:1875] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:37.700258 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 250445us
W20240503 02:12:37.700714 4746 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.700886 4746 leader_election.cc:302] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.701017 6057 consensus_queue.cc:1035] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.701436 6057 raft_consensus.cc:2726] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.702235 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cf5d681047e040968c0436492ac29ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:37.702893 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf5d681047e040968c0436492ac29ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.704459 4271 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 876) took 1518 ms. Trace:
I20240503 02:12:37.704576 4271 rpcz_store.cc:276] 0503 02:12:36.186089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.939942 (+753853us) service_pool.cc:225] Handling call
0503 02:12:36.941134 (+ 1192us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:36.941718 (+ 584us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.704162 (+762444us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.704294 (+ 132us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.704395 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:36.976584 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:36.976711 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:36.976718 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:36.976720 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:36.976830 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:36.976843 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:36.976849 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:36.976971 (+ 122us) tablet.cc:699] Row locks acquired
0503 02:12:36.976974 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:36.977016 (+ 42us) write_op.cc:270] Start()
0503 02:12:36.977054 (+ 38us) write_op.cc:276] Timestamp: P: 1714702356976991 usec, L: 0
0503 02:12:36.977057 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:36.977253 (+ 196us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.701802 (+724549us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.702155 (+ 353us) write_op.cc:301] APPLY: starting
0503 02:12:37.702175 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.702231 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.702279 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.702428 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.702469 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.702477 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.702560 (+ 83us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.702630 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.702653 (+ 23us) write_op.cc:454] Released schema lock
0503 02:12:37.702661 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.702720 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.702724 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":290,"apply.run_cpu_time_us":629,"apply.run_wall_time_us":628,"num_ops":2,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":778,"prepare.run_wall_time_us":778,"raft.queue_time_us":432,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":724714}]]}
I20240503 02:12:37.705720 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6b15997a94b74ad3808d965acfb5d155
I20240503 02:12:37.706279 4269 catalog_manager.cc:6554] ReplaceTablet: tablet 6b15997a94b74ad3808d965acfb5d155 deleted and replaced by tablet f11a1f9872714a3a91b1179e0d44a14a
I20240503 02:12:37.706456 4269 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 873) took 1717 ms. Trace:
I20240503 02:12:37.706708 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b15997a94b74ad3808d965acfb5d155 with delete_type TABLET_DATA_DELETED (replaced by tablet f11a1f9872714a3a91b1179e0d44a14a) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.706544 4269 rpcz_store.cc:276] 0503 02:12:35.988877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.862474 (+873597us) service_pool.cc:225] Handling call
0503 02:12:37.706365 (+843891us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.020203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.020344 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.020350 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:37.020353 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:37.020494 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.020506 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.020512 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.020597 (+ 85us) tablet.cc:699] Row locks acquired
0503 02:12:37.020600 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.020639 (+ 39us) write_op.cc:270] Start()
0503 02:12:37.020674 (+ 35us) write_op.cc:276] Timestamp: P: 1714702357020611 usec, L: 0
0503 02:12:37.020678 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.056114 (+ 35436us) log.cc:842] Serialized 622 byte log entry
0503 02:12:37.056510 (+ 396us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:37.702010 (+645500us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.702806 (+ 796us) write_op.cc:301] APPLY: starting
0503 02:12:37.702816 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.702855 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.702859 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.702974 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.703032 (+ 58us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:37.703041 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:37.703134 (+ 93us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.703192 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.703213 (+ 21us) write_op.cc:454] Released schema lock
0503 02:12:37.703223 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.703273 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.703276 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":759,"apply.run_cpu_time_us":511,"apply.run_wall_time_us":510,"num_ops":2,"prepare.queue_time_us":1174,"prepare.run_cpu_time_us":1736,"prepare.run_wall_time_us":36383,"raft.queue_time_us":1119,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":6,"replication_time_us":681312,"spinlock_wait_cycles":80945920}]]}
I20240503 02:12:37.707609 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d84412b28b4c41c2a3ce18702403dcd8
I20240503 02:12:37.707768 5105 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:37.708181 4266 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 868) took 1822 ms. Trace:
I20240503 02:12:37.708560 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.708276 4266 rpcz_store.cc:276] 0503 02:12:35.885802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.849903 (+964101us) service_pool.cc:225] Handling call
0503 02:12:36.850377 (+ 474us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:36.850519 (+ 142us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:36.850582 (+ 63us) catalog_manager.cc:2758] Locking tablets
0503 02:12:37.019301 (+168719us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:37.707430 (+688129us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:37.707510 (+ 80us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:37.707556 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:37.708115 (+ 559us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.056641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.056806 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.056814 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:37.056816 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.056989 (+ 173us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.057004 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.057011 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.057131 (+ 120us) tablet.cc:699] Row locks acquired
0503 02:12:37.057134 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.057171 (+ 37us) write_op.cc:270] Start()
0503 02:12:37.057215 (+ 44us) write_op.cc:276] Timestamp: P: 1714702357057142 usec, L: 0
0503 02:12:37.057219 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.057422 (+ 203us) log.cc:842] Serialized 725 byte log entry
0503 02:12:37.702078 (+644656us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.703347 (+ 1269us) write_op.cc:301] APPLY: starting
0503 02:12:37.703357 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.703390 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.703414 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.703477 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.703502 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:37.703509 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:37.703557 (+ 48us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.703594 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.703607 (+ 13us) write_op.cc:454] Released schema lock
0503 02:12:37.703612 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.703654 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.703658 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1240,"apply.run_cpu_time_us":348,"apply.run_wall_time_us":347,"num_ops":2,"prepare.queue_time_us":36497,"prepare.run_cpu_time_us":876,"prepare.run_wall_time_us":875,"raft.queue_time_us":216,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":644841}]]}
I20240503 02:12:37.714375 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:37.714807 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:37.715117 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:37.715874 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2288fa2ae0674a20b78e456480dd514b" } new_table_name: "default.Table_f9768db1fb854c20803813eba4596d3b" modify_external_catalogs: true
I20240503 02:12:37.717640 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ab8976cbfe464bc88d520662b34937ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:37.718271 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab8976cbfe464bc88d520662b34937ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.720018 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ef7f7e5ca4de4d209727d3bdf577ce9f" } new_table_name: "default.Table_9b1a0366eab9453e9bb829ff9d3a51f2" modify_external_catalogs: true
I20240503 02:12:37.720209 5159 ts_tablet_manager.cc:1888] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.720757 5159 log.cc:1196] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/396b48865e954d05b9ba8084a2e7f29b
W20240503 02:12:37.722456 4273 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 86) took 773 ms (client timeout 1000 ms). Trace:
I20240503 02:12:37.722529 5159 ts_tablet_manager.cc:1909] T 396b48865e954d05b9ba8084a2e7f29b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:37.722641 4273 rpcz_store.cc:269] 0503 02:12:36.948523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.021859 (+ 73336us) service_pool.cc:225] Handling call
0503 02:12:37.056620 (+ 34761us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.722390 (+665770us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":76627200}
I20240503 02:12:37.723177 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_42acc9ebe0264f3a833dcf672cf1b830"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:37.723824 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42acc9ebe0264f3a833dcf672cf1b830 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.724159 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5c3f0c84a0db4405ad8fed6849923e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:37.723982 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 581) took 4346 ms. Trace:
W20240503 02:12:37.724637 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c3f0c84a0db4405ad8fed6849923e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.724581 5159 rpcz_store.cc:276] 0503 02:12:33.377815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.379956 (+ 2141us) service_pool.cc:225] Handling call
0503 02:12:37.556485 (+4176529us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:37.556497 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.682537 (+126040us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.719475 (+ 36938us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.723709 (+ 4234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4162639}
I20240503 02:12:37.726042 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_07f00cb83ef24520a8b4e944ab24b09f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:37.726055 5159 tablet_replica.cc:330] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:37.726521 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07f00cb83ef24520a8b4e944ab24b09f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.727208 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:37.727396 4270 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 100) took 1385 ms (client timeout 1000 ms). Trace:
I20240503 02:12:37.727427 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.727597 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:37.727522 4270 rpcz_store.cc:269] 0503 02:12:36.341824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.735420 (+393596us) service_pool.cc:225] Handling call
0503 02:12:36.744661 (+ 9241us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.727345 (+982684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.057547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.057644 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.057650 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:37.057652 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.057760 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.057770 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.057774 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.057863 (+ 89us) tablet.cc:699] Row locks acquired
0503 02:12:37.057866 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.057896 (+ 30us) write_op.cc:270] Start()
0503 02:12:37.057924 (+ 28us) write_op.cc:276] Timestamp: P: 1714702357057869 usec, L: 0
0503 02:12:37.057928 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.058132 (+ 204us) log.cc:842] Serialized 901 byte log entry
0503 02:12:37.704193 (+646061us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.704289 (+ 96us) write_op.cc:301] APPLY: starting
0503 02:12:37.704300 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.704332 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.704360 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.704410 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.704451 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:37.704459 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.704529 (+ 70us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.704590 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.704611 (+ 21us) write_op.cc:454] Released schema lock
0503 02:12:37.704617 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.704669 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.704673 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":30,"spinlock_wait_cycles":18464128,"child_traces":[["op",{"apply.queue_time_us":35,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":423,"num_ops":2,"prepare.queue_time_us":27654,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":660,"raft.queue_time_us":68,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":646246,"spinlock_wait_cycles":58112}]]}
I20240503 02:12:37.727952 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:37.733376 5105 ts_tablet_manager.cc:1367] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.685s user 0.008s sys 0.008s
I20240503 02:12:37.733320 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_78930698028a4f4ab89c3beaf915f453"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:37.733397 5159 raft_consensus.cc:2229] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:37.734014 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78930698028a4f4ab89c3beaf915f453 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.734421 5105 raft_consensus.cc:348] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.734658 5105 raft_consensus.cc:374] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.734719 5105 raft_consensus.cc:729] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.734927 5105 consensus_queue.cc:260] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.735095 5159 raft_consensus.cc:2258] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.735515 5105 ts_tablet_manager.cc:1398] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:37.735828 5105 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:37.737529 4746 leader_election.cc:341] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.738010 5159 ts_tablet_manager.cc:1875] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.739872 5103 tablet_bootstrap.cc:492] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:37.740063 5103 ts_tablet_manager.cc:1367] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.211s user 0.005s sys 0.004s
I20240503 02:12:37.741092 5103 raft_consensus.cc:348] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.741374 5103 raft_consensus.cc:374] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.741472 5103 raft_consensus.cc:729] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.741845 5103 consensus_queue.cc:260] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.744540 5103 ts_tablet_manager.cc:1398] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:37.747960 5103 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:37.756711 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4dac17dca3442d8934ac357779b78a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.757647 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.757880 4805 raft_consensus.cc:2454] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:37.759184 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.759423 4805 raft_consensus.cc:2379] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:37.760205 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:37.760381 4805 raft_consensus.cc:3037] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.763545 4805 raft_consensus.cc:2454] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:37.764184 6353 consensus_queue.cc:1035] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.780441 5105 tablet_bootstrap.cc:654] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:37.782783 6247 raft_consensus.cc:482] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:37.783556 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 111525us
I20240503 02:12:37.783618 5999 raft_consensus.cc:2726] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 02:12:37.783941 4608 leader_election.cc:341] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.783919 5999 consensus_queue.cc:1046] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:12:37.784148 4608 leader_election.cc:341] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.784184 5999 raft_consensus.cc:3032] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:37.784245 5999 raft_consensus.cc:729] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:37.653297 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152f205cf1b4ec7ae03605bbfa007e1" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:37.784936 6317 consensus_queue.cc:1046] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:37.784440 5999 consensus_queue.cc:260] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.782967 6247 raft_consensus.cc:504] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.789417 5999 raft_consensus.cc:3037] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:37.789850 6247 leader_election.cc:288] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:37.790251 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ef3c6e5e7d40249b112144975ba220" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:37.790409 4667 raft_consensus.cc:3037] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.792762 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed5d1c5bb054ac6ac2edf9ea03ba5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.796907 5103 tablet_bootstrap.cc:654] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:37.802369 4746 leader_election.cc:341] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.802834 4746 leader_election.cc:302] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:37.804751 6349 raft_consensus.cc:2726] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:37.806743 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8debae3d66514458b2f2ac04982975b8 (table default.Table_6518e9cc40e1483f885638cd1a931f6a [id=dfdaada115ed4a4daac3fcfe1a9bb0aa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8debae3d66514458b2f2ac04982975b8 consensus state has no leader
I20240503 02:12:37.805456 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.809420 4746 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:37.809749 6353 raft_consensus.cc:2781] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:37.809990 6353 raft_consensus.cc:686] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:37.810297 6353 consensus_queue.cc:237] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.816016 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 396b48865e954d05b9ba8084a2e7f29b (table default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]) successfully deleted
I20240503 02:12:37.816344 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb59bbf7b68431c86d88f27ef2528b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.820643 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e80bdc70d45e8ae0c7bb162654868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.822638 4274 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 880) took 1636 ms. Trace:
I20240503 02:12:37.823046 4274 rpcz_store.cc:276] 0503 02:12:36.186469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.461333 (+1274864us) service_pool.cc:225] Handling call
0503 02:12:37.702115 (+240782us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.703001 (+ 886us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:37.703512 (+ 511us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.822330 (+118818us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.822489 (+ 159us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.822591 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.805364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.805522 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.805531 (+ 9us) write_op.cc:435] Acquired schema lock
0503 02:12:37.805534 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:37.805672 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.805688 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.805693 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.805801 (+ 108us) tablet.cc:699] Row locks acquired
0503 02:12:37.805805 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:37.805849 (+ 44us) write_op.cc:270] Start()
0503 02:12:37.805889 (+ 40us) write_op.cc:276] Timestamp: P: 1714702357805645 usec, L: 0
0503 02:12:37.805896 (+ 7us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.806149 (+ 253us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.818900 (+ 12751us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.819039 (+ 139us) write_op.cc:301] APPLY: starting
0503 02:12:37.819059 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.819113 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.819165 (+ 52us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.819303 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.819350 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.819360 (+ 10us) write_op.cc:312] APPLY: finished
0503 02:12:37.819456 (+ 96us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.819526 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.819551 (+ 25us) write_op.cc:454] Released schema lock
0503 02:12:37.819584 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.819654 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.819661 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":552925056,"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":681,"apply.run_wall_time_us":682,"num_ops":2,"prepare.queue_time_us":101327,"prepare.run_cpu_time_us":979,"prepare.run_wall_time_us":7696,"raft.queue_time_us":395,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":303,"replication_time_us":12974,"thread_start_us":109909,"threads_started":1}]]}
I20240503 02:12:37.823148 5115 tablet_replica.cc:330] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:37.827667 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ba631d85854b43bcb4bece9bfba7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.830080 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82190dfb1b1418b9f20bf4d0c2c2409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.832716 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b15997a94b74ad3808d965acfb5d155 with delete_type TABLET_DATA_DELETED (replaced by tablet f11a1f9872714a3a91b1179e0d44a14a) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.833214 6296 consensus_queue.cc:1035] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.837257 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.818320 4462 leader_election.cc:302] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:37.839038 4462 leader_election.cc:302] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:37.839361 5868 raft_consensus.cc:3037] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.840435 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.840905 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.842075 5868 raft_consensus.cc:2726] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.842358 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.842468 5868 raft_consensus.cc:3037] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.842799 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.843237 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:37.845002 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30d115c3aabf484e88a3d55377038182 consensus state has no leader
I20240503 02:12:37.845183 4272 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 878) took 1658 ms. Trace:
I20240503 02:12:37.845214 4462 leader_election.cc:302] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:37.845499 6183 raft_consensus.cc:2781] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.845597 6183 raft_consensus.cc:686] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:37.845291 4272 rpcz_store.cc:276] 0503 02:12:36.186292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.427821 (+1241529us) service_pool.cc:225] Handling call
0503 02:12:37.715792 (+287971us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.716102 (+ 310us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:37.716355 (+ 253us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:37.716576 (+ 221us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:37.716662 (+ 86us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:37.844676 (+128014us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:37.845152 (+ 476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.813106 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.813282 (+ 176us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.813290 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:37.813293 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:37.814493 (+ 1200us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.814506 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.814511 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:37.816381 (+ 1870us) tablet.cc:699] Row locks acquired
0503 02:12:37.816384 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.817890 (+ 1506us) write_op.cc:270] Start()
0503 02:12:37.817941 (+ 51us) write_op.cc:276] Timestamp: P: 1714702357816224 usec, L: 0
0503 02:12:37.817947 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.825016 (+ 7069us) log.cc:842] Serialized 405 byte log entry
0503 02:12:37.826415 (+ 1399us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:37.843654 (+ 17239us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.843936 (+ 282us) write_op.cc:301] APPLY: starting
0503 02:12:37.843956 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.844018 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.844020 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.844078 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.844117 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:37.844125 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:37.844242 (+ 117us) log.cc:842] Serialized 23 byte log entry
0503 02:12:37.844304 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.844326 (+ 22us) write_op.cc:454] Released schema lock
0503 02:12:37.844334 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.844390 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.844393 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":661627136,"child_traces":[["op",{"apply.queue_time_us":225,"apply.run_cpu_time_us":525,"apply.run_wall_time_us":525,"num_ops":1,"prepare.queue_time_us":95838,"prepare.run_cpu_time_us":1407,"prepare.run_wall_time_us":14542,"raft.queue_time_us":632,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":309,"replication_time_us":25674,"spinlock_wait_cycles":14652160}]]}
I20240503 02:12:37.845849 6183 consensus_queue.cc:237] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.846127 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_61c9ba66e720449da2d10423b0c029bb" } modify_external_catalogs: true
I20240503 02:12:37.846333 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_61c9ba66e720449da2d10423b0c029bb" } modify_external_catalogs: true
I20240503 02:12:37.847254 6294 log.cc:824] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:37.853741 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.856784 6183 raft_consensus.cc:482] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.856978 6183 raft_consensus.cc:504] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.857579 6183 leader_election.cc:288] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.863375 5868 raft_consensus.cc:2726] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:37.864761 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.865556 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.866091 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.884088 6349 consensus_queue.cc:1035] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.886641 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 8750b02c5936412db525d2d3e77befeb with delete_type TABLET_DATA_DELETED (replaced by tablet 1f3dbcbc572f4215b7f59f3750a4b9fa) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.887044 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:37.888578 4269 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 882) took 1689 ms. Trace:
I20240503 02:12:37.888732 4269 rpcz_store.cc:276] 0503 02:12:36.199174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.707156 (+1507982us) service_pool.cc:225] Handling call
0503 02:12:37.717502 (+ 10346us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.718382 (+ 880us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:37.718989 (+ 607us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.851889 (+132900us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.888337 (+ 36448us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.888482 (+ 145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.827678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.827824 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.827831 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:37.827834 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:37.827963 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.827976 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.827983 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.828083 (+ 100us) tablet.cc:699] Row locks acquired
0503 02:12:37.828088 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:37.828131 (+ 43us) write_op.cc:270] Start()
0503 02:12:37.828182 (+ 51us) write_op.cc:276] Timestamp: P: 1714702357827923 usec, L: 0
0503 02:12:37.828187 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.828377 (+ 190us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.850819 (+ 22442us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.851105 (+ 286us) write_op.cc:301] APPLY: starting
0503 02:12:37.851130 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.851220 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.851258 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.851402 (+ 144us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.851457 (+ 55us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.851469 (+ 12us) write_op.cc:312] APPLY: finished
0503 02:12:37.851570 (+ 101us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.851651 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.851681 (+ 30us) write_op.cc:454] Released schema lock
0503 02:12:37.851692 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.851764 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.851771 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":22979328,"child_traces":[["op",{"apply.queue_time_us":219,"apply.run_cpu_time_us":724,"apply.run_wall_time_us":725,"num_ops":2,"prepare.queue_time_us":108150,"prepare.run_cpu_time_us":805,"prepare.run_wall_time_us":10734,"replication_time_us":22602}]]}
W20240503 02:12:37.889767 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4f2c4c285c14857a7403a74d55f11ee consensus state has no leader
I20240503 02:12:37.889961 4267 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36560 (request call id 879) took 1703 ms. Trace:
I20240503 02:12:37.891428 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.890604 4267 rpcz_store.cc:276] 0503 02:12:36.186361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.459836 (+1273475us) service_pool.cc:225] Handling call
0503 02:12:37.719939 (+260103us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7aaf46 00007fd3f179dc26 00007fd3f179d955 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.720228 (+ 289us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:37.720454 (+ 226us) catalog_manager.cc:3613] Apply alter schema
0503 02:12:37.720625 (+ 171us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 02:12:37.720717 (+ 92us) catalog_manager.cc:3783] Updating metadata on disk
0503 02:12:37.857789 (+137072us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 02:12:37.889895 (+ 32106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.838464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.838585 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.838592 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:37.838594 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.838709 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.838722 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.838727 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:37.838787 (+ 60us) tablet.cc:699] Row locks acquired
0503 02:12:37.838790 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.838827 (+ 37us) write_op.cc:270] Start()
0503 02:12:37.838858 (+ 31us) write_op.cc:276] Timestamp: P: 1714702357838621 usec, L: 0
0503 02:12:37.838862 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.839041 (+ 179us) log.cc:842] Serialized 405 byte log entry
0503 02:12:37.855821 (+ 16780us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.856201 (+ 380us) write_op.cc:301] APPLY: starting
0503 02:12:37.856224 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.856277 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.856327 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.856390 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.856436 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:37.856448 (+ 12us) write_op.cc:312] APPLY: finished
0503 02:12:37.856538 (+ 90us) log.cc:842] Serialized 23 byte log entry
0503 02:12:37.856598 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.856620 (+ 22us) write_op.cc:454] Released schema lock
0503 02:12:37.856628 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.856680 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.856683 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":597540096,"child_traces":[["op",{"apply.queue_time_us":302,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":537,"num_ops":1,"prepare.queue_time_us":117251,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":15746,"replication_time_us":16931}]]}
I20240503 02:12:37.892815 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_04015aedd8a14fecbad557d674b75f31" } new_table_name: "default.Table_003ced126c26439d80b384dd00c1b8a7" modify_external_catalogs: true
I20240503 02:12:37.894709 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d372fe582377472fb9b1ae98f230a96f" } modify_external_catalogs: true
I20240503 02:12:37.894913 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d372fe582377472fb9b1ae98f230a96f" } modify_external_catalogs: true
I20240503 02:12:37.895298 6057 raft_consensus.cc:482] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.895462 6057 raft_consensus.cc:504] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.896239 6057 leader_election.cc:288] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.898618 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.901307 4271 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36560 (request call id 881) took 1713 ms. Trace:
I20240503 02:12:37.901863 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 19fb0929bd234788b5313d96d2e4845c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.901425 4271 rpcz_store.cc:276] 0503 02:12:36.187377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.705256 (+1517879us) service_pool.cc:225] Handling call
0503 02:12:37.724039 (+ 18783us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a8cb3 00007fd3f179a2a6 00007fd3f1799fd5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.724719 (+ 680us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:12:37.725150 (+ 431us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:12:37.900998 (+175848us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:12:37.901141 (+ 143us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:12:37.901236 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.863347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.863559 (+ 212us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.863590 (+ 31us) write_op.cc:435] Acquired schema lock
0503 02:12:37.863592 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.863788 (+ 196us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.863809 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.863818 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.864005 (+ 187us) tablet.cc:699] Row locks acquired
0503 02:12:37.864009 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:37.864081 (+ 72us) write_op.cc:270] Start()
0503 02:12:37.864234 (+ 153us) write_op.cc:276] Timestamp: P: 1714702357863868 usec, L: 0
0503 02:12:37.864240 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.890478 (+ 26238us) log.cc:842] Serialized 433 byte log entry
0503 02:12:37.900352 (+ 9874us) op_driver.cc:444] REPLICATION: finished
0503 02:12:37.900469 (+ 117us) write_op.cc:301] APPLY: starting
0503 02:12:37.900482 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:37.900529 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:37.900571 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:37.900688 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:37.900727 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:37.900736 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:12:37.900808 (+ 72us) log.cc:842] Serialized 29 byte log entry
0503 02:12:37.900872 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.900893 (+ 21us) write_op.cc:454] Released schema lock
0503 02:12:37.900900 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:37.900978 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:37.900982 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":42493824,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":559,"apply.run_wall_time_us":558,"num_ops":2,"prepare.queue_time_us":128865,"prepare.run_cpu_time_us":1365,"prepare.run_wall_time_us":34345,"raft.queue_time_us":218,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":293,"replication_time_us":36085}]]}
I20240503 02:12:37.902609 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_453e312025324be6a4d15ae5116a46fa" } modify_external_catalogs: true
I20240503 02:12:37.902798 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_453e312025324be6a4d15ae5116a46fa" } modify_external_catalogs: true
I20240503 02:12:37.902899 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:37.903662 6057 raft_consensus.cc:482] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.903820 6057 raft_consensus.cc:504] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:37.904883 6057 leader_election.cc:288] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:37.908484 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94fb9d53044d796e2973143d888fb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:37.908476 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f042afb954daa923502ef12377bd5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:37.910687 4746 leader_election.cc:341] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.912439 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8fa27e930c344572afb5f6ba668633df" } modify_external_catalogs: true
I20240503 02:12:37.912664 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8fa27e930c344572afb5f6ba668633df" } modify_external_catalogs: true
I20240503 02:12:37.914528 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1c52b1bbf50148c5b4753da4f8fe72fc" } new_table_name: "default.Table_de11f8796c984bd79ffadd090787b9a5" modify_external_catalogs: true
W20240503 02:12:37.917480 4746 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:37.929275 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_34db9afaf7fc4588903d0918bcbb98a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:37.929944 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34db9afaf7fc4588903d0918bcbb98a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.931950 5114 ts_tablet_manager.cc:1888] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.932168 5114 log.cc:1196] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:37.932766 5114 ts_tablet_manager.cc:1909] T 22776c81d1a24d368abb68274cf32a0b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:37.933378 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 584) took 4216 ms. Trace:
I20240503 02:12:37.933568 5114 rpcz_store.cc:276] 0503 02:12:33.717176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.717243 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:37.643398 (+3926155us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:37.643408 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.684255 (+ 40847us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x616000230254. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139eb8ba00 00007f139fe65b83 00007f13a28f957c 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74 00007f139336719b 00007f1393367018
0503 02:12:37.731911 (+ 47656us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.931860 (+199949us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.933354 (+ 1494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59979136,"tablet-delete.queue_time_us":3925725}
I20240503 02:12:37.934321 5114 tablet_replica.cc:330] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:37.934314 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 22776c81d1a24d368abb68274cf32a0b (table default.Table_98ecda629a0e48c3bcb69d91efbfe2f4 [id=07ea56ce8d3749dfbcbc1a79ea84693c]) successfully deleted
I20240503 02:12:37.937193 5159 ts_tablet_manager.cc:1888] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:37.937325 5159 log.cc:1196] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5dbbfa2d1a49400c809d5b476036cdcd
I20240503 02:12:37.937743 5159 ts_tablet_manager.cc:1909] T 5dbbfa2d1a49400c809d5b476036cdcd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:37.938504 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 591) took 4515 ms. Trace:
I20240503 02:12:37.938585 5159 rpcz_store.cc:276] 0503 02:12:33.422604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.422666 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:37.725846 (+4303180us) ts_tablet_manager.cc:1120] Deleting tablet 5dbbfa2d1a49400c809d5b476036cdcd
0503 02:12:37.725857 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.792083 (+ 66226us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.898498 (+106415us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:37.938482 (+ 39984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4302707}
I20240503 02:12:37.939265 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 5dbbfa2d1a49400c809d5b476036cdcd (table default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]) successfully deleted
W20240503 02:12:37.939765 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c084ab3fedb46d4992f5af2ceede06b consensus state has no leader
I20240503 02:12:37.940256 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5d2a8af7b53948258ce2c6f305f318cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:37.940871 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d2a8af7b53948258ce2c6f305f318cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:37.943087 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 (DEFAULT_TABLE table=default.Table_9714c86da62e48489868fed877679b05 [id=ab7e568df1ab4c918307ae4fefcd9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.943504 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab2b01fb13c4aaa931b3a633ab5d847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.943496 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 81017bd22de44ac59354a40e35966bcf (DEFAULT_TABLE table=default.Table_d5f2e666523c426c8f54c9f677fbdd3c [id=831eda426e9d43e3a3d2469de1b63a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.943833 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81017bd22de44ac59354a40e35966bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.947515 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 81017bd22de44ac59354a40e35966bcf (DEFAULT_TABLE table=default.Table_d5f2e666523c426c8f54c9f677fbdd3c [id=831eda426e9d43e3a3d2469de1b63a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.947860 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81017bd22de44ac59354a40e35966bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.950299 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c2b47b4153d841e6bf89131335cdb1c2 (DEFAULT_TABLE table=default.Table_5f22a82442be4541886bad9e8ada3df9 [id=035010094fcf4e45b38502b23eed6c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.950639 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b47b4153d841e6bf89131335cdb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.953294 4491 tablet_service.cc:1467] Processing CreateTablet for tablet c2b47b4153d841e6bf89131335cdb1c2 (DEFAULT_TABLE table=default.Table_5f22a82442be4541886bad9e8ada3df9 [id=035010094fcf4e45b38502b23eed6c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.953672 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b47b4153d841e6bf89131335cdb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.948874 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 592) took 4521 ms. Trace:
I20240503 02:12:37.954237 5159 rpcz_store.cc:276] 0503 02:12:33.427268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.427326 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:37.948752 (+4521426us) ts_tablet_manager.cc:1120] Deleting tablet 857d019b8ab94be1b7719fec83e57e55
0503 02:12:37.948761 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.948856 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4521268}
W20240503 02:12:37.954739 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 857d019b8ab94be1b7719fec83e57e55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857d019b8ab94be1b7719fec83e57e55 already in progress: creating tablet
I20240503 02:12:37.955260 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 594) took 4454 ms. Trace:
I20240503 02:12:37.955336 5159 rpcz_store.cc:276] 0503 02:12:33.500593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.500660 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:37.955125 (+4454465us) ts_tablet_manager.cc:1120] Deleting tablet 16fcfe6e2dc844a486476bf4be80d5cb
0503 02:12:37.955135 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.955240 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4454202}
W20240503 02:12:37.955750 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 16fcfe6e2dc844a486476bf4be80d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fcfe6e2dc844a486476bf4be80d5cb already in progress: creating tablet
I20240503 02:12:37.957201 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 (DEFAULT_TABLE table=default.Table_9714c86da62e48489868fed877679b05 [id=ab7e568df1ab4c918307ae4fefcd9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:37.957558 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab2b01fb13c4aaa931b3a633ab5d847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:37.960476 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 597) took 4364 ms. Trace:
I20240503 02:12:37.960558 5159 rpcz_store.cc:276] 0503 02:12:33.596485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.597095 (+ 610us) service_pool.cc:225] Handling call
0503 02:12:37.960350 (+4363255us) ts_tablet_manager.cc:1120] Deleting tablet dee34be89bda450296b916163bb10c46
0503 02:12:37.960360 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:37.960457 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4362992}
W20240503 02:12:37.961047 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet dee34be89bda450296b916163bb10c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee34be89bda450296b916163bb10c46 already in progress: creating tablet
I20240503 02:12:37.965140 5159 tablet_replica.cc:330] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:37.967650 5159 raft_consensus.cc:2229] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:37.969923 4667 raft_consensus.cc:2454] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:37.971832 5159 raft_consensus.cc:2258] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:37.975898 5159 ts_tablet_manager.cc:1875] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:37.976807 4274 catalog_manager.cc:5484] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:37.977567 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b14241dd76864e8a93d0279d3ab8f3ac (table default.Table_c5ef79d34f424f519e36c63857a7cce9 [id=713930e2953343799de0e9c465e1c083]). Expected version 1 got 0
I20240503 02:12:37.982270 4810 raft_consensus.cc:1261] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:12:37.982911 5103 tablet_bootstrap.cc:492] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:37.983471 5868 consensus_queue.cc:1035] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:37.993399 5103 ts_tablet_manager.cc:1367] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.247s user 0.005s sys 0.000s
I20240503 02:12:37.993752 4520 raft_consensus.cc:3037] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.994218 5103 raft_consensus.cc:348] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.994438 5103 raft_consensus.cc:374] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:37.994505 5103 raft_consensus.cc:729] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:37.994706 5103 consensus_queue.cc:260] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:37.995244 5103 ts_tablet_manager.cc:1398] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:37.995551 5103 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:37.996960 5103 tablet_bootstrap.cc:654] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:37.998260 6353 consensus_queue.cc:1035] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.007441 5105 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.007643 5105 ts_tablet_manager.cc:1367] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.272s user 0.005s sys 0.001s
I20240503 02:12:38.008885 5105 raft_consensus.cc:348] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.009133 5105 raft_consensus.cc:374] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.009233 5105 raft_consensus.cc:729] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.009480 5105 consensus_queue.cc:260] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.010000 5105 ts_tablet_manager.cc:1398] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:38.010298 5105 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.011605 5105 tablet_bootstrap.cc:654] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.014742 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:38.031250 5114 raft_consensus.cc:2229] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.033928 5114 raft_consensus.cc:2258] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:38.035818 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 250808us
W20240503 02:12:38.040416 4608 leader_election.cc:341] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.041064 6353 raft_consensus.cc:482] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.041244 6353 raft_consensus.cc:504] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.041913 6353 leader_election.cc:288] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.046068 4608 leader_election.cc:302] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.046010 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.046479 6262 raft_consensus.cc:2781] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.046775 6262 raft_consensus.cc:686] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:38.047042 5103 tablet_bootstrap.cc:492] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.047070 6262 consensus_queue.cc:237] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:38.047533 4746 leader_election.cc:341] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.048019 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94fb9d53044d796e2973143d888fb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.050628 5103 ts_tablet_manager.cc:1367] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20240503 02:12:38.051024 4608 leader_election.cc:302] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.052881 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5c40f1f6b40308eed26b64d047494" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.053196 4674 raft_consensus.cc:2379] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate acc6d33969e74c209310005ae911c405 in this term.
I20240503 02:12:38.053524 5115 raft_consensus.cc:2229] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.053719 5115 raft_consensus.cc:2258] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.054316 5115 ts_tablet_manager.cc:1875] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.054658 6247 raft_consensus.cc:3032] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:12:38.054741 6247 raft_consensus.cc:729] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:38.055640 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f042afb954daa923502ef12377bd5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.056696 5103 raft_consensus.cc:348] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.057094 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:38.057109 4746 leader_election.cc:341] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.057307 4746 leader_election.cc:302] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.057427 5103 raft_consensus.cc:374] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.057785 6296 raft_consensus.cc:2726] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.057770 4462 leader_election.cc:302] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.058055 6183 raft_consensus.cc:2726] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.058331 6296 consensus_queue.cc:1035] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.058405 5103 raft_consensus.cc:729] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.058708 5103 consensus_queue.cc:260] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.059415 5103 ts_tablet_manager.cc:1398] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:38.059785 5103 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.060067 6296 consensus_queue.cc:1035] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:38.060740 4746 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.060889 4746 leader_election.cc:302] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.061283 6296 raft_consensus.cc:2726] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:38.061748 4746 leader_election.cc:341] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.061937 4746 leader_election.cc:302] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.062165 6057 raft_consensus.cc:2726] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.062561 5114 ts_tablet_manager.cc:1875] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:38.070165 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3b9341be62a4275ad6e4c7d7b745621 consensus state has no leader
I20240503 02:12:38.070601 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ab5717cf4ced4b728fad84ef05751e14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:38.071210 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab5717cf4ced4b728fad84ef05751e14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.077180 5103 tablet_bootstrap.cc:654] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.079061 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0e89c1cf6ed54d2fafb2173ea9373ab5
I20240503 02:12:38.080175 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8f8b6599206549dd8ddf449a52c61fce
I20240503 02:12:38.081907 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8b6599206549dd8ddf449a52c61fce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.067620 4667 raft_consensus.cc:1261] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:12:38.077828 5999 raft_consensus.cc:2781] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.082983 5999 raft_consensus.cc:686] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:38.083240 5999 consensus_queue.cc:237] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.083773 6400 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:38.078084 4608 leader_election.cc:302] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.085165 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.085368 5851 raft_consensus.cc:2781] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.085438 5851 raft_consensus.cc:482] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.085503 5851 raft_consensus.cc:3037] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.086156 5105 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.086308 5105 ts_tablet_manager.cc:1367] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.006s
I20240503 02:12:38.086619 4608 leader_election.cc:302] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.087152 6382 raft_consensus.cc:2741] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:12:38.087203 5105 raft_consensus.cc:348] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.087397 5105 raft_consensus.cc:374] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.087450 5105 raft_consensus.cc:729] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.087630 5105 consensus_queue.cc:260] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.088342 5105 ts_tablet_manager.cc:1398] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:38.088801 5105 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.090629 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8b6599206549dd8ddf449a52c61fce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.091190 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.091248 5105 tablet_bootstrap.cc:654] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.097695 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:38.097920 4810 raft_consensus.cc:2454] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:38.098237 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 09005fb01c384122a7310c1eea12d3f0 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:38.099835 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.108672 6400 tablet.cc:1722] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.112536 5159 ts_tablet_manager.cc:1888] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.113305 5159 log.cc:1196] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:38.116325 6013 consensus_queue.cc:1035] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.117656 5159 ts_tablet_manager.cc:1909] T 22776c81d1a24d368abb68274cf32a0b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.119097 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 598) took 4405 ms. Trace:
I20240503 02:12:38.119287 5159 rpcz_store.cc:276] 0503 02:12:33.713377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.713634 (+ 257us) service_pool.cc:225] Handling call
0503 02:12:37.965070 (+4251436us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:37.965083 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.082049 (+116966us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.111893 (+ 29844us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.119045 (+ 7152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4250948}
I20240503 02:12:38.120375 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 22776c81d1a24d368abb68274cf32a0b (table default.Table_98ecda629a0e48c3bcb69d91efbfe2f4 [id=07ea56ce8d3749dfbcbc1a79ea84693c]) successfully deleted
I20240503 02:12:38.121166 5159 tablet_replica.cc:330] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.121546 5159 raft_consensus.cc:2229] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.121892 5159 raft_consensus.cc:2258] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.121976 6400 tablet.cc:1722] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:38.122989 5159 ts_tablet_manager.cc:1875] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.133411 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet d84412b28b4c41c2a3ce18702403dcd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:36 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.134145 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b15997a94b74ad3808d965acfb5d155 with delete_type TABLET_DATA_DELETED (replaced by tablet f11a1f9872714a3a91b1179e0d44a14a) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:38.134217 6395 log.cc:924] Time spent T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240503 02:12:38.136229 4636 tablet_service.cc:1467] Processing CreateTablet for tablet c2b47b4153d841e6bf89131335cdb1c2 (DEFAULT_TABLE table=default.Table_5f22a82442be4541886bad9e8ada3df9 [id=035010094fcf4e45b38502b23eed6c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.136382 5115 ts_tablet_manager.cc:1888] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.136466 5115 log.cc:1196] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/617ad14927c1441b800a204aefc2e619
I20240503 02:12:38.136639 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b47b4153d841e6bf89131335cdb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.136802 5115 ts_tablet_manager.cc:1909] T 617ad14927c1441b800a204aefc2e619 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:38.139073 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 81017bd22de44ac59354a40e35966bcf (DEFAULT_TABLE table=default.Table_d5f2e666523c426c8f54c9f677fbdd3c [id=831eda426e9d43e3a3d2469de1b63a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.138912 6407 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:38.055006 6247 consensus_queue.cc:260] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.087903 5851 raft_consensus.cc:504] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.145788 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.148061 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 577) took 4725 ms. Trace:
I20240503 02:12:38.148622 5115 rpcz_store.cc:276] 0503 02:12:33.422957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.423455 (+ 498us) service_pool.cc:225] Handling call
0503 02:12:37.815266 (+4391811us) ts_tablet_manager.cc:1120] Deleting tablet 617ad14927c1441b800a204aefc2e619
0503 02:12:37.818717 (+ 3451us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.084636 (+265919us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.136275 (+ 51639us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.148032 (+ 11757us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":225927,"tablet-delete.queue_time_us":4391591}
I20240503 02:12:38.149838 5115 tablet_replica.cc:330] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:38.150995 5115 raft_consensus.cc:2229] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.151867 5115 raft_consensus.cc:2258] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.152848 5115 ts_tablet_manager.cc:1875] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.161623 5105 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.161798 5105 ts_tablet_manager.cc:1367] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.003s
I20240503 02:12:38.165369 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 617ad14927c1441b800a204aefc2e619 (table default.Table_27d1ed70e95c4b648e63e37a036f1b42 [id=909b67c764cf482b89844bd9cc50127b]) successfully deleted
I20240503 02:12:38.165567 5105 raft_consensus.cc:348] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.165781 5105 raft_consensus.cc:374] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.165840 5105 raft_consensus.cc:729] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.167598 5105 consensus_queue.cc:260] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.168033 5105 ts_tablet_manager.cc:1398] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:38.168232 5159 ts_tablet_manager.cc:1888] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:38.168812 5159 log.cc:1196] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:38.168320 5105 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.174011 5159 ts_tablet_manager.cc:1909] T 78b21a3010a4470288fbdd684ef64e77 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.176986 5105 tablet_bootstrap.cc:654] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.177219 6407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51106 (request call id 524) took 9098 ms. Trace:
I20240503 02:12:38.180049 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 616) took 4056 ms. Trace:
I20240503 02:12:38.177843 6407 rpcz_store.cc:276] 0503 02:12:29.078226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:31.524671 (+2446445us) service_pool.cc:225] Handling call
0503 02:12:38.177020 (+6652349us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:31.642867 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:31.643671 (+ 804us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:31.643683 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:31.643692 (+ 9us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:31.643740 (+ 48us) alter_schema_op.cc:119] START. Timestamp: 1714702351643706
0503 02:12:31.643745 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:31.644263 (+ 518us) log.cc:842] Serialized 173 byte log entry
0503 02:12:38.070183 (+6425920us) op_driver.cc:444] REPLICATION: finished
0503 02:12:38.138428 (+ 68245us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:38.143943 (+ 5515us) log.cc:842] Serialized 5 byte log entry
0503 02:12:38.174524 (+ 30581us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.176140 (+ 1616us) log.cc:842] Serialized 14 byte log entry
0503 02:12:38.176596 (+ 456us) alter_schema_op.cc:65] Released schema lock
0503 02:12:38.176767 (+ 171us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67937,"prepare.queue_time_us":117328,"prepare.run_cpu_time_us":5568,"prepare.run_wall_time_us":31668,"replication_time_us":6426411,"thread_start_us":144700,"threads_started":3,"wal-append.queue_time_us":29497,"wal-append.run_cpu_time_us":538,"wal-append.run_wall_time_us":1000588}]]}
I20240503 02:12:38.180478 5159 rpcz_store.cc:276] 0503 02:12:34.123531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.123590 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:38.121080 (+3997490us) ts_tablet_manager.cc:1120] Deleting tablet 78b21a3010a4470288fbdd684ef64e77
0503 02:12:38.121093 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.136405 (+ 15312us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.167810 (+ 31405us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.179759 (+ 11949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3996344}
I20240503 02:12:38.183030 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 78b21a3010a4470288fbdd684ef64e77 (table default.Table_7566b876f57547599a98c4f55fbfc427 [id=5d52bb117988409d974ea961f9f7bab9]) successfully deleted
I20240503 02:12:38.185084 6407 tablet.cc:1722] T 7d919b9262ac48cbb8b3f893918b5590 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:38.185518 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 617) took 3948 ms. Trace:
I20240503 02:12:38.185982 5159 rpcz_store.cc:276] 0503 02:12:34.236857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.236930 (+ 73us) service_pool.cc:225] Handling call
0503 02:12:38.184573 (+3947643us) ts_tablet_manager.cc:1120] Deleting tablet 211dfd4049684f5e8af927b2a19263b7
0503 02:12:38.184741 (+ 168us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.185353 (+ 612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3947424}
W20240503 02:12:38.187736 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 211dfd4049684f5e8af927b2a19263b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211dfd4049684f5e8af927b2a19263b7 already in progress: creating tablet
I20240503 02:12:38.189298 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 618) took 3878 ms. Trace:
I20240503 02:12:38.189770 5159 rpcz_store.cc:276] 0503 02:12:34.311321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.315476 (+ 4155us) service_pool.cc:225] Handling call
0503 02:12:38.188772 (+3873296us) ts_tablet_manager.cc:1120] Deleting tablet 50e04292742646049f39b6aa29338d75
0503 02:12:38.188783 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.189276 (+ 493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3872779}
W20240503 02:12:38.191149 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 50e04292742646049f39b6aa29338d75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e04292742646049f39b6aa29338d75 already in progress: creating tablet
I20240503 02:12:38.154711 6375 consensus_queue.cc:1046] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:38.192526 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 619) took 3852 ms. Trace:
I20240503 02:12:38.192708 6247 raft_consensus.cc:3037] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:38.192788 5159 rpcz_store.cc:276] 0503 02:12:34.339855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.339945 (+ 90us) service_pool.cc:225] Handling call
0503 02:12:38.191951 (+3852006us) ts_tablet_manager.cc:1120] Deleting tablet 6e635f283288406aa16b3941be64e777
0503 02:12:38.191965 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.192413 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851617}
W20240503 02:12:38.193774 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6e635f283288406aa16b3941be64e777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e635f283288406aa16b3941be64e777 already in progress: creating tablet
I20240503 02:12:38.194046 6296 raft_consensus.cc:482] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.194167 6296 raft_consensus.cc:504] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.194794 6296 leader_election.cc:288] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.195307 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.194766 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 620) took 3853 ms. Trace:
W20240503 02:12:38.195816 4746 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.155622 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 (DEFAULT_TABLE table=default.Table_9714c86da62e48489868fed877679b05 [id=ab7e568df1ab4c918307ae4fefcd9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.195540 5159 rpcz_store.cc:276] 0503 02:12:34.341627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.341700 (+ 73us) service_pool.cc:225] Handling call
0503 02:12:38.194239 (+3852539us) ts_tablet_manager.cc:1120] Deleting tablet c3710b653aa24b7ca389aa95d7cdd2fe
0503 02:12:38.194248 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.194747 (+ 499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3852310}
I20240503 02:12:38.165956 5851 leader_election.cc:288] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.170789 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8b6599206549dd8ddf449a52c61fce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:38.196794 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c3710b653aa24b7ca389aa95d7cdd2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3710b653aa24b7ca389aa95d7cdd2fe already in progress: creating tablet
I20240503 02:12:38.180023 6411 tablet.cc:1722] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.189710 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81017bd22de44ac59354a40e35966bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.197508 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab2b01fb13c4aaa931b3a633ab5d847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.197806 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 621) took 3627 ms. Trace:
I20240503 02:12:38.198500 5159 rpcz_store.cc:276] 0503 02:12:34.570459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.570554 (+ 95us) service_pool.cc:225] Handling call
0503 02:12:38.197519 (+3626965us) ts_tablet_manager.cc:1120] Deleting tablet f5fd27023b3f47f281dfb5474ea3a4c6
0503 02:12:38.197529 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.197785 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3622663}
W20240503 02:12:38.200485 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f5fd27023b3f47f281dfb5474ea3a4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5fd27023b3f47f281dfb5474ea3a4c6 already in progress: creating tablet
I20240503 02:12:38.201443 6296 consensus_queue.cc:1035] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.201661 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 628) took 3457 ms. Trace:
I20240503 02:12:38.202032 5159 rpcz_store.cc:276] 0503 02:12:34.743834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.744061 (+ 227us) service_pool.cc:225] Handling call
0503 02:12:38.201155 (+3457094us) ts_tablet_manager.cc:1120] Deleting tablet 84f6279f16dc42d69e794e3c6586163d
0503 02:12:38.201260 (+ 105us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.201551 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3456752}
I20240503 02:12:38.204154 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 629) took 3460 ms. Trace:
I20240503 02:12:38.204447 5159 rpcz_store.cc:276] 0503 02:12:34.743929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.744078 (+ 149us) service_pool.cc:225] Handling call
0503 02:12:38.203632 (+3459554us) ts_tablet_manager.cc:1120] Deleting tablet 04d555c8d8844ee49c960a5279485480
0503 02:12:38.203646 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.204044 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3459050}
W20240503 02:12:38.205753 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 04d555c8d8844ee49c960a5279485480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04d555c8d8844ee49c960a5279485480 already in progress: creating tablet
I20240503 02:12:38.206745 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 626) took 3463 ms. Trace:
I20240503 02:12:38.207167 5159 rpcz_store.cc:276] 0503 02:12:34.743594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.745686 (+ 2092us) service_pool.cc:225] Handling call
0503 02:12:38.206333 (+3460647us) ts_tablet_manager.cc:1120] Deleting tablet b8474c841cb64373a223ad76b43ecf5f
0503 02:12:38.206345 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.206724 (+ 379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460296}
W20240503 02:12:38.208387 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b8474c841cb64373a223ad76b43ecf5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8474c841cb64373a223ad76b43ecf5f already in progress: creating tablet
I20240503 02:12:38.209529 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 627) took 3465 ms. Trace:
I20240503 02:12:38.209877 5159 rpcz_store.cc:276] 0503 02:12:34.743732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746104 (+ 2372us) service_pool.cc:225] Handling call
0503 02:12:38.209018 (+3462914us) ts_tablet_manager.cc:1120] Deleting tablet a8ce4b33f90c40c48cf0a538a1a7fde4
0503 02:12:38.209032 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.209419 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462549}
W20240503 02:12:38.210927 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ce4b33f90c40c48cf0a538a1a7fde4 already in progress: creating tablet
I20240503 02:12:38.212519 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 630) took 3468 ms. Trace:
I20240503 02:12:38.212909 5159 rpcz_store.cc:276] 0503 02:12:34.744033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746408 (+ 2375us) service_pool.cc:225] Handling call
0503 02:12:38.212126 (+3465718us) ts_tablet_manager.cc:1120] Deleting tablet 8135907b78284d7ca74396cc77dd0612
0503 02:12:38.212230 (+ 104us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.212500 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465472}
W20240503 02:12:38.214028 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8135907b78284d7ca74396cc77dd0612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8135907b78284d7ca74396cc77dd0612 already in progress: creating tablet
I20240503 02:12:38.215081 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 631) took 3418 ms. Trace:
I20240503 02:12:38.215536 5159 rpcz_store.cc:276] 0503 02:12:34.797051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.797300 (+ 249us) service_pool.cc:225] Handling call
0503 02:12:38.214794 (+3417494us) ts_tablet_manager.cc:1120] Deleting tablet 7447242714c844e685faca5aed71a7f7
0503 02:12:38.214805 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.215065 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3417045}
W20240503 02:12:38.216584 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7447242714c844e685faca5aed71a7f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7447242714c844e685faca5aed71a7f7 already in progress: creating tablet
I20240503 02:12:38.216629 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.216866 5868 consensus_queue.cc:1035] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:38.217586 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 632) took 3394 ms. Trace:
I20240503 02:12:38.217967 5159 rpcz_store.cc:276] 0503 02:12:34.822696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.822894 (+ 198us) service_pool.cc:225] Handling call
0503 02:12:38.217090 (+3394196us) ts_tablet_manager.cc:1120] Deleting tablet cef84b4e6ba24150905994d45ffd0b52
0503 02:12:38.217104 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.217562 (+ 458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3393877}
W20240503 02:12:38.219240 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet cef84b4e6ba24150905994d45ffd0b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef84b4e6ba24150905994d45ffd0b52 already in progress: creating tablet
I20240503 02:12:38.220090 5159 tablet_replica.cc:330] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:38.221004 4746 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.221123 4746 leader_election.cc:302] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.221678 6296 raft_consensus.cc:2726] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.221040 5159 raft_consensus.cc:2229] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.221673 5868 consensus_queue.cc:1035] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.222608 5159 raft_consensus.cc:2258] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.225356 5159 ts_tablet_manager.cc:1875] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.229470 5106 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:38.229648 5106 ts_tablet_manager.cc:1367] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.794s user 0.002s sys 0.010s
I20240503 02:12:38.240273 6296 raft_consensus.cc:482] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.240437 6296 raft_consensus.cc:504] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.241142 6296 leader_election.cc:288] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.242002 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178760c21904207b6c8317093871f63" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:38.242564 4746 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.252844 5159 ts_tablet_manager.cc:1888] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:38.253698 5159 log.cc:1196] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/17afd88bf6c040ad9e8d571eccd348dc
I20240503 02:12:38.252774 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178760c21904207b6c8317093871f63" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.255725 5159 ts_tablet_manager.cc:1909] T 17afd88bf6c040ad9e8d571eccd348dc P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.259780 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 633) took 3162 ms. Trace:
I20240503 02:12:38.260303 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 56aa7303f7784d2289a8da9653d1e317
I20240503 02:12:38.261003 4272 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 886) took 1196 ms. Trace:
I20240503 02:12:38.261134 4272 rpcz_store.cc:276] 0503 02:12:37.064091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.845965 (+781874us) service_pool.cc:225] Handling call
0503 02:12:37.846454 (+ 489us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:37.846718 (+ 264us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:37.846791 (+ 73us) catalog_manager.cc:2758] Locking tablets
0503 02:12:37.942077 (+ 95286us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:38.260100 (+318023us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:38.260199 (+ 99us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:38.260245 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:38.260900 (+ 655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.954786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.954861 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.954866 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:37.954868 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.954950 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.954957 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.954961 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.955038 (+ 77us) tablet.cc:699] Row locks acquired
0503 02:12:37.955040 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:37.955064 (+ 24us) write_op.cc:270] Start()
0503 02:12:37.955085 (+ 21us) write_op.cc:276] Timestamp: P: 1714702357954852 usec, L: 0
0503 02:12:37.955088 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.955206 (+ 118us) log.cc:842] Serialized 725 byte log entry
0503 02:12:38.258944 (+303738us) op_driver.cc:444] REPLICATION: finished
0503 02:12:38.259087 (+ 143us) write_op.cc:301] APPLY: starting
0503 02:12:38.259103 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:38.259153 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:38.259203 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:38.259274 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:38.259314 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:38.259321 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:38.259398 (+ 77us) log.cc:842] Serialized 29 byte log entry
0503 02:12:38.259473 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.259491 (+ 18us) write_op.cc:454] Released schema lock
0503 02:12:38.259498 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:38.259556 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.259583 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":550,"num_ops":2,"prepare.queue_time_us":12213,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":461,"replication_time_us":303831}]]}
I20240503 02:12:38.261950 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 56aa7303f7784d2289a8da9653d1e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.262578 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 20b6a18be64049049e1a083bf4af51ba
I20240503 02:12:38.263084 4267 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 888) took 1186 ms. Trace:
I20240503 02:12:38.263213 4267 rpcz_store.cc:276] 0503 02:12:37.076724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.891963 (+815239us) service_pool.cc:225] Handling call
0503 02:12:37.894636 (+ 2673us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a9ca6 00007fd3f179bf66 00007fd3f179bc95 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:37.895041 (+ 405us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:37.895166 (+ 125us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:37.895217 (+ 51us) catalog_manager.cc:2758] Locking tablets
0503 02:12:37.962935 (+ 67718us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:38.262350 (+299415us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:38.262450 (+ 100us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:38.262507 (+ 57us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:38.263046 (+ 539us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.969022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:37.969168 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:37.969174 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:37.969176 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:37.969617 (+ 441us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:37.969630 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:37.969635 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:37.970055 (+ 420us) tablet.cc:699] Row locks acquired
0503 02:12:37.970058 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:37.970100 (+ 42us) write_op.cc:270] Start()
0503 02:12:37.970138 (+ 38us) write_op.cc:276] Timestamp: P: 1714702357969881 usec, L: 0
0503 02:12:37.970142 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.970331 (+ 189us) log.cc:842] Serialized 725 byte log entry
0503 02:12:38.259019 (+288688us) op_driver.cc:444] REPLICATION: finished
0503 02:12:38.259682 (+ 663us) write_op.cc:301] APPLY: starting
0503 02:12:38.259692 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:38.259726 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:38.259755 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:38.259804 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:38.259831 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:38.259839 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:38.259891 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 02:12:38.259943 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.259956 (+ 13us) write_op.cc:454] Released schema lock
0503 02:12:38.259962 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:38.260002 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.260005 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5547392,"child_traces":[["op",{"apply.queue_time_us":620,"apply.run_cpu_time_us":373,"apply.run_wall_time_us":372,"num_ops":2,"prepare.queue_time_us":5374,"prepare.run_cpu_time_us":829,"prepare.run_wall_time_us":2630,"replication_time_us":288864}]]}
I20240503 02:12:38.263753 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b6a18be64049049e1a083bf4af51ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.264364 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b6a18be64049049e1a083bf4af51ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.264609 5115 ts_tablet_manager.cc:1888] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.260376 5159 rpcz_store.cc:276] 0503 02:12:35.097520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.097598 (+ 78us) service_pool.cc:225] Handling call
0503 02:12:38.219937 (+3122339us) ts_tablet_manager.cc:1120] Deleting tablet 17afd88bf6c040ad9e8d571eccd348dc
0503 02:12:38.219950 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.239325 (+ 19375us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.252451 (+ 13126us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.259745 (+ 7294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3121936}
I20240503 02:12:38.265326 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 56aa7303f7784d2289a8da9653d1e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.265851 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 17afd88bf6c040ad9e8d571eccd348dc (table default.Table_8b93d7e7be90441ea960cdf3b81e4fc7 [id=5febc608abd84e7ba26288f2e155d449]) successfully deleted
I20240503 02:12:38.267072 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 634) took 2981 ms. Trace:
I20240503 02:12:38.267504 5159 rpcz_store.cc:276] 0503 02:12:35.285498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.285557 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:38.266537 (+2980980us) ts_tablet_manager.cc:1120] Deleting tablet 6ecdb634fb1c4337b2eae0c844fbb1dc
0503 02:12:38.266553 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.266960 (+ 407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2980649}
I20240503 02:12:38.270004 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 635) took 2978 ms. Trace:
I20240503 02:12:38.270884 5159 rpcz_store.cc:276] 0503 02:12:35.291637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.291708 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:38.269207 (+2977499us) ts_tablet_manager.cc:1120] Deleting tablet ee1fac803dd84eee8722f0a14559faf6
0503 02:12:38.269221 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.269826 (+ 605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977072}
I20240503 02:12:38.272601 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
W20240503 02:12:38.273965 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecdb634fb1c4337b2eae0c844fbb1dc already in progress: creating tablet
W20240503 02:12:38.274240 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ee1fac803dd84eee8722f0a14559faf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1fac803dd84eee8722f0a14559faf6 already in progress: creating tablet
I20240503 02:12:38.274703 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 636) took 2971 ms. Trace:
I20240503 02:12:38.275287 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.275523 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.275398 5159 rpcz_store.cc:276] 0503 02:12:35.302829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.302886 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:38.273349 (+2970463us) ts_tablet_manager.cc:1120] Deleting tablet 9290a61c21fa456eb3c4f9b0dbebc864
0503 02:12:38.273526 (+ 177us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.274460 (+ 934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2970254}
I20240503 02:12:38.276688 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.277774 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:38.277827 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9290a61c21fa456eb3c4f9b0dbebc864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9290a61c21fa456eb3c4f9b0dbebc864 already in progress: creating tablet
I20240503 02:12:38.279911 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 650) took 2223 ms. Trace:
I20240503 02:12:38.280676 5159 rpcz_store.cc:276] 0503 02:12:36.056905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.056979 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:38.278797 (+2221818us) ts_tablet_manager.cc:1120] Deleting tablet 3e94a9c61cb74cc08ca5867ae85ecdf0
0503 02:12:38.279043 (+ 246us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.279889 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2218556}
W20240503 02:12:38.283779 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 already in progress: creating tablet
I20240503 02:12:38.284852 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.285969 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:38.286648 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:38.286931 4275 catalog_manager.cc:5484] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 1 to 2, leader changed from f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65) to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 2 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.287597 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:38.288056 4275 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 7d919b9262ac48cbb8b3f893918b5590 (table default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95]). Expected version 3 got 0
I20240503 02:12:38.290474 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 648) took 2233 ms. Trace:
I20240503 02:12:38.291085 5159 rpcz_store.cc:276] 0503 02:12:36.056500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.056551 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:38.285645 (+2229094us) ts_tablet_manager.cc:1120] Deleting tablet 8c6c80efe35343989f46a2c181f38129
0503 02:12:38.289493 (+ 3848us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.290104 (+ 611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225503}
W20240503 02:12:38.293279 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8c6c80efe35343989f46a2c181f38129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6c80efe35343989f46a2c181f38129 already in progress: creating tablet
I20240503 02:12:38.295503 5159 tablet_replica.cc:330] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.297029 5159 raft_consensus.cc:2229] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.298079 5159 raft_consensus.cc:2258] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:38.298498 4746 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.298624 4746 leader_election.cc:302] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.299190 6353 raft_consensus.cc:2726] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.299362 6353 consensus_queue.cc:1035] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.300902 5159 ts_tablet_manager.cc:1875] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.311466 5115 log.cc:1196] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:38.311911 5115 ts_tablet_manager.cc:1909] T 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:38.312242 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 578) took 4889 ms. Trace:
I20240503 02:12:38.312320 5115 rpcz_store.cc:276] 0503 02:12:33.423113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.423699 (+ 586us) service_pool.cc:225] Handling call
0503 02:12:38.149775 (+4726076us) ts_tablet_manager.cc:1120] Deleting tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
0503 02:12:38.149789 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.235696 (+ 85907us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.264500 (+ 28804us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.312222 (+ 47722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4725889}
I20240503 02:12:38.312698 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 579) took 4885 ms. Trace:
I20240503 02:12:38.312768 5115 rpcz_store.cc:276] 0503 02:12:33.427208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.427265 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:38.312577 (+4885312us) ts_tablet_manager.cc:1120] Deleting tablet 857d019b8ab94be1b7719fec83e57e55
0503 02:12:38.312586 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.312684 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4885095}
I20240503 02:12:38.316551 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ab8976cbfe464bc88d520662b34937ab" } modify_external_catalogs: true
I20240503 02:12:38.316923 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8fbb7708f7674d1f86c2b5e38808225b" } new_table_name: "default.Table_5bb43ae60af64383b89a700d96f6fad7" modify_external_catalogs: true
I20240503 02:12:38.316982 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1afe5e17ba0045cfb7cc930692d02e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.317207 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cb6f41cdb8344d96ba919d9493b64365"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.317749 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ab8976cbfe464bc88d520662b34937ab" } modify_external_catalogs: true
W20240503 02:12:38.318009 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb6f41cdb8344d96ba919d9493b64365 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.318153 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9313c394ecf4460a81267e309fdd6748" } new_table_name: "default.Table_c5703a4668b04accbd9b3f93427b31ab" modify_external_catalogs: true
I20240503 02:12:38.319531 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c5ef79d34f424f519e36c63857a7cce9" } new_table_name: "default.Table_918295a06e0d4e09af7c72203e0c3448" modify_external_catalogs: true
W20240503 02:12:38.320451 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1afe5e17ba0045cfb7cc930692d02e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.320611 4496 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51106 (request call id 688) took 1102 ms. Trace:
I20240503 02:12:38.320701 4496 rpcz_store.cc:276] 0503 02:12:37.218041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.218102 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:37.444640 (+226538us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:12:37.444676 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:12:38.065905 (+621229us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.320440 (+254535us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:38.320586 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:38.322986 5105 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.323176 5105 ts_tablet_manager.cc:1367] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.017s
I20240503 02:12:38.323931 5105 raft_consensus.cc:348] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.315192 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 580) took 4822 ms. Trace:
I20240503 02:12:38.324223 5115 rpcz_store.cc:276] 0503 02:12:33.493155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.493218 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:38.315085 (+4821867us) ts_tablet_manager.cc:1120] Deleting tablet 16fcfe6e2dc844a486476bf4be80d5cb
0503 02:12:38.315094 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.315174 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4821580}
I20240503 02:12:38.324445 5105 raft_consensus.cc:374] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.324534 5105 raft_consensus.cc:729] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:38.325135 6410 log.cc:924] Time spent T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240503 02:12:38.325132 5105 consensus_queue.cc:260] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.329666 5115 tablet_replica.cc:330] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:38.329937 5115 raft_consensus.cc:2229] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.330255 5115 raft_consensus.cc:2258] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.319290 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3755b2f2996b4bc785bf23c4c7af2cf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:38.333678 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3755b2f2996b4bc785bf23c4c7af2cf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.336159 5115 ts_tablet_manager.cc:1875] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.330804 5159 ts_tablet_manager.cc:1888] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:38.339440 5159 log.cc:1196] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:38.340600 5159 ts_tablet_manager.cc:1909] T 65774b69a8bd4664a6b55b84eba37f7f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.342170 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 651) took 2285 ms. Trace:
I20240503 02:12:38.342325 5159 rpcz_store.cc:276] 0503 02:12:36.057103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.057144 (+ 41us) service_pool.cc:225] Handling call
0503 02:12:38.295100 (+2237956us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:38.295284 (+ 184us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.312487 (+ 17203us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.328684 (+ 16197us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.342116 (+ 13432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2234865}
I20240503 02:12:38.339883 5105 ts_tablet_manager.cc:1398] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:12:38.345144 5105 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:38.346197 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 181005us
W20240503 02:12:38.346464 4608 leader_election.cc:341] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.347901 4608 leader_election.cc:302] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.348167 6317 raft_consensus.cc:2781] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.348241 6317 raft_consensus.cc:482] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.348310 6317 raft_consensus.cc:3037] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.349380 6393 consensus_queue.cc:1035] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.350574 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:38.350772 4808 raft_consensus.cc:3037] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.350922 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:38.353932 4808 raft_consensus.cc:2454] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:38.355273 6407 tablet.cc:1722] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.356156 5105 tablet_bootstrap.cc:654] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.356694 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 649) took 2300 ms. Trace:
I20240503 02:12:38.359122 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a5d6a733e83e43b39bf0c003e569df5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.357012 5159 rpcz_store.cc:276] 0503 02:12:36.056724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.056763 (+ 39us) service_pool.cc:225] Handling call
0503 02:12:38.343044 (+2286281us) ts_tablet_manager.cc:1120] Deleting tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2
0503 02:12:38.343117 (+ 73us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.356646 (+ 13529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277876}
I20240503 02:12:38.360531 5159 tablet_replica.cc:330] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.361029 5159 raft_consensus.cc:2229] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.361263 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 65774b69a8bd4664a6b55b84eba37f7f (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
I20240503 02:12:38.361536 5159 raft_consensus.cc:2258] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.365389 5159 ts_tablet_manager.cc:1875] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.366091 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c (table default.Table_a210808781ce413f96c49311edbe1415 [id=4a5fa9cb182a4b18b902652020d89d00]) successfully deleted
I20240503 02:12:38.369541 5103 tablet_bootstrap.cc:492] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.369740 5103 ts_tablet_manager.cc:1367] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.310s user 0.034s sys 0.008s
I20240503 02:12:38.370653 5103 raft_consensus.cc:348] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.371531 5103 raft_consensus.cc:374] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.371634 5103 raft_consensus.cc:729] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.371894 5103 consensus_queue.cc:260] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.372527 5103 ts_tablet_manager.cc:1398] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 02:12:38.372874 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5d6a733e83e43b39bf0c003e569df5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:38.373019 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 857d019b8ab94be1b7719fec83e57e55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857d019b8ab94be1b7719fec83e57e55 already in progress: creating tablet
W20240503 02:12:38.373737 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 16fcfe6e2dc844a486476bf4be80d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fcfe6e2dc844a486476bf4be80d5cb already in progress: creating tablet
W20240503 02:12:38.373999 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 already in progress: creating tablet
I20240503 02:12:38.383862 5103 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.385933 5103 tablet_bootstrap.cc:654] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.388294 5868 raft_consensus.cc:482] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.388455 5868 raft_consensus.cc:504] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.389127 5868 leader_election.cc:288] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.389478 5105 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.389719 5105 ts_tablet_manager.cc:1367] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20240503 02:12:38.389902 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:38.390169 4808 raft_consensus.cc:2379] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
W20240503 02:12:38.390913 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf67d8c1d3ec4f758cca9745c42c7ee6 consensus state has no leader
I20240503 02:12:38.391397 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_73dc468aacee46c397690de4bffb0c0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.392901 5159 ts_tablet_manager.cc:1888] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.393504 5159 log.cc:1196] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ea4dac17dca3442d8934ac357779b78a
W20240503 02:12:38.395110 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73dc468aacee46c397690de4bffb0c0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.396363 5159 ts_tablet_manager.cc:1909] T ea4dac17dca3442d8934ac357779b78a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.396971 4266 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 320a53ce75644322ac6f78b3cb9b5e0f
W20240503 02:12:38.398331 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9470480a9de45209044f8e8c461519a consensus state has no leader
I20240503 02:12:38.396747 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2184d913525e433d87c1fc3346faf9d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.400231 5114 ts_tablet_manager.cc:1888] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:38.400312 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2184d913525e433d87c1fc3346faf9d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.401621 5105 raft_consensus.cc:348] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.405354 6317 raft_consensus.cc:504] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.406075 6317 leader_election.cc:288] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.405097 5105 raft_consensus.cc:374] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.406951 5105 raft_consensus.cc:729] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.408385 6411 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:57782 (request call id 673) took 1333 ms. Trace:
I20240503 02:12:38.408085 5105 consensus_queue.cc:260] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.409108 5105 ts_tablet_manager.cc:1398] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:12:38.409392 5105 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.411105 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 652) took 2353 ms. Trace:
I20240503 02:12:38.411494 5159 rpcz_store.cc:276] 0503 02:12:36.057234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.057276 (+ 42us) service_pool.cc:225] Handling call
0503 02:12:38.360445 (+2303169us) ts_tablet_manager.cc:1120] Deleting tablet ea4dac17dca3442d8934ac357779b78a
0503 02:12:38.360458 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.379491 (+ 19033us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.392527 (+ 13036us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.410986 (+ 18459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292852}
I20240503 02:12:38.413534 5159 tablet_replica.cc:330] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.414747 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ea4dac17dca3442d8934ac357779b78a (table default.Table_55c37198f9d2480784f643011b70dee8 [id=89060d0da21a4cf8b3d3c1ab1a8a3749]) successfully deleted
I20240503 02:12:38.414868 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cea9de5ffb3e411d90226d13e4eb39b5" } new_table_name: "default.Table_9e181c295fb749eaa9445ce83763c3e8" modify_external_catalogs: true
I20240503 02:12:38.416585 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b70b33ab1fd34982a6f9463002af9bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:38.417260 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b70b33ab1fd34982a6f9463002af9bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.417668 5114 log.cc:1196] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/53017352b8f34955b116bbfd32f04854
I20240503 02:12:38.418077 5114 ts_tablet_manager.cc:1909] T 53017352b8f34955b116bbfd32f04854 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:38.420334 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 56aa7303f7784d2289a8da9653d1e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.420969 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b6a18be64049049e1a083bf4af51ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.422391 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:38.422605 4808 raft_consensus.cc:3037] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.419363 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c2a8190210439fb7589cb02ec82bc2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.426153 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7461aa76a1af41cb9253285bda708f30" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:38.427196 4462 leader_election.cc:302] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.427739 5868 raft_consensus.cc:3037] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.433547 5868 raft_consensus.cc:2726] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.434445 4808 raft_consensus.cc:2454] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:38.443261 5103 tablet_bootstrap.cc:492] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.444092 5103 ts_tablet_manager.cc:1367] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
W20240503 02:12:38.445889 4608 consensus_peers.cc:485] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22776c81d1a24d368abb68274cf32a0b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:38.449784 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_76c5d3de08164c7c893ba2fe01b1e38b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:38.450428 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76c5d3de08164c7c893ba2fe01b1e38b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:38.451323 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a8e996d76be4dda876f4645df56c1ce consensus state has no leader
I20240503 02:12:38.451685 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_96ec56aa54e84e89a80c091d6bcd5e2f" } modify_external_catalogs: true
I20240503 02:12:38.451907 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_96ec56aa54e84e89a80c091d6bcd5e2f" } modify_external_catalogs: true
I20240503 02:12:38.451984 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4cc509b619474ecd8fc3f145f028ce24" } new_table_name: "default.Table_4a987b64a137496cb458d34c67236398" modify_external_catalogs: true
I20240503 02:12:38.452526 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8e334557d1ca4d7faa91def038f0afc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.453410 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_27e6472c5d7746f59ee6399fa6db4d3b" } modify_external_catalogs: true
W20240503 02:12:38.453518 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e334557d1ca4d7faa91def038f0afc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.453606 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_27e6472c5d7746f59ee6399fa6db4d3b" } modify_external_catalogs: true
I20240503 02:12:38.448077 5103 raft_consensus.cc:348] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.455536 5103 raft_consensus.cc:374] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.456905 5103 raft_consensus.cc:729] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.457350 5103 consensus_queue.cc:260] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.458782 5106 raft_consensus.cc:348] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.458969 5106 raft_consensus.cc:374] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.459018 5106 raft_consensus.cc:729] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.460073 5103 ts_tablet_manager.cc:1398] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:38.462189 5105 tablet_bootstrap.cc:654] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.449045 5159 raft_consensus.cc:2229] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.462952 5159 raft_consensus.cc:2258] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.446187 4608 leader_election.cc:302] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:38.467350 6393 raft_consensus.cc:2781] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.467487 6393 raft_consensus.cc:686] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:38.467914 4608 leader_election.cc:302] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:38.468163 6317 raft_consensus.cc:2781] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.468261 6317 raft_consensus.cc:686] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:38.468739 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b013c448c7de4b10bc993479d2bd7cd8" } modify_external_catalogs: true
I20240503 02:12:38.468983 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b013c448c7de4b10bc993479d2bd7cd8" } modify_external_catalogs: true
W20240503 02:12:38.469374 4608 leader_election.cc:341] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.470515 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 602) took 4625 ms. Trace:
I20240503 02:12:38.471014 5114 rpcz_store.cc:276] 0503 02:12:33.844838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.849097 (+ 4259us) service_pool.cc:225] Handling call
0503 02:12:37.934242 (+4085145us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:37.934252 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.241011 (+306759us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.400069 (+159058us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.470479 (+ 70410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4084843}
I20240503 02:12:38.471638 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 53017352b8f34955b116bbfd32f04854 (table default.Table_bb18c51dac8c46b2a5ddecca6ddd34be [id=47856b6b36f74b5589f9acc60162c1e8]) successfully deleted
I20240503 02:12:38.470945 5103 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.473980 5159 ts_tablet_manager.cc:1875] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.474511 5103 tablet_bootstrap.cc:654] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.477054 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 603) took 4345 ms. Trace:
I20240503 02:12:38.477707 5114 rpcz_store.cc:276] 0503 02:12:34.131365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.131481 (+ 116us) service_pool.cc:225] Handling call
0503 02:12:38.475730 (+4344249us) ts_tablet_manager.cc:1120] Deleting tablet c3710b653aa24b7ca389aa95d7cdd2fe
0503 02:12:38.475743 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.477031 (+ 1288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4339323}
I20240503 02:12:38.478881 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e800e34461a64783bd120662f37d31a3" } modify_external_catalogs: true
W20240503 02:12:38.478951 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c3710b653aa24b7ca389aa95d7cdd2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3710b653aa24b7ca389aa95d7cdd2fe already in progress: creating tablet
I20240503 02:12:38.479115 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e800e34461a64783bd120662f37d31a3" } modify_external_catalogs: true
I20240503 02:12:38.481019 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 604) took 4348 ms. Trace:
I20240503 02:12:38.482277 5114 rpcz_store.cc:276] 0503 02:12:34.132868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.219861 (+ 86993us) service_pool.cc:225] Handling call
0503 02:12:38.478195 (+4258334us) ts_tablet_manager.cc:1120] Deleting tablet 78b21a3010a4470288fbdd684ef64e77
0503 02:12:38.478205 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.478303 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4258053}
I20240503 02:12:38.481197 5106 consensus_queue.cc:260] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.484010 6317 consensus_queue.cc:237] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.484436 6393 consensus_queue.cc:237] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.408480 6411 rpcz_store.cc:276] 0503 02:12:37.075250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.081092 (+ 5842us) service_pool.cc:225] Handling call
0503 02:12:38.408348 (+1327256us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:37.082033 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 02:12:37.082151 (+ 118us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 02:12:37.082160 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0503 02:12:37.082167 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 02:12:37.082201 (+ 34us) alter_schema_op.cc:119] START. Timestamp: 1714702357082141
0503 02:12:37.082204 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:37.082374 (+ 170us) log.cc:842] Serialized 173 byte log entry
0503 02:12:38.091225 (+1008851us) op_driver.cc:444] REPLICATION: finished
0503 02:12:38.179993 (+ 88768us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 02:12:38.199838 (+ 19845us) log.cc:842] Serialized 5 byte log entry
0503 02:12:38.339876 (+140038us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.408268 (+ 68392us) log.cc:842] Serialized 14 byte log entry
0503 02:12:38.408311 (+ 43us) alter_schema_op.cc:65] Released schema lock
0503 02:12:38.408315 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88679,"prepare.queue_time_us":576,"prepare.run_cpu_time_us":2167,"prepare.run_wall_time_us":7636,"replication_time_us":1008990,"thread_start_us":105009,"threads_started":3,"wal-append.queue_time_us":7357}]]}
I20240503 02:12:38.493961 6411 tablet.cc:1722] T 7d919b9262ac48cbb8b3f893918b5590 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:38.498492 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:38.502885 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet da12dd28e03f462b905d1844e2ad3340
I20240503 02:12:38.490006 4520 raft_consensus.cc:3037] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:38.508443 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c27e37ebd0f49cdba7653548acbe1ea consensus state has no leader
I20240503 02:12:38.511026 6353 consensus_queue.cc:1035] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.512141 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet da12dd28e03f462b905d1844e2ad3340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.513137 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet da12dd28e03f462b905d1844e2ad3340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:38.514374 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 605) took 4276 ms. Trace:
I20240503 02:12:38.514464 5114 rpcz_store.cc:276] 0503 02:12:34.237543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.237600 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:38.514193 (+4276593us) ts_tablet_manager.cc:1120] Deleting tablet 211dfd4049684f5e8af927b2a19263b7
0503 02:12:38.514205 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.514354 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4275484}
I20240503 02:12:38.522956 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 606) took 4284 ms. Trace:
I20240503 02:12:38.523545 5114 rpcz_store.cc:276] 0503 02:12:34.238287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.238349 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:38.522792 (+4284443us) ts_tablet_manager.cc:1120] Deleting tablet 50e04292742646049f39b6aa29338d75
0503 02:12:38.522806 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.522933 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4284183}
W20240503 02:12:38.526783 4275 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.66:56851 (request call id 86) took 1576 ms (client timeout 1000 ms). Trace:
I20240503 02:12:38.528095 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 607) took 4187 ms. Trace:
I20240503 02:12:38.528712 5114 rpcz_store.cc:276] 0503 02:12:34.340807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.340881 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:38.527934 (+4187053us) ts_tablet_manager.cc:1120] Deleting tablet 6e635f283288406aa16b3941be64e777
0503 02:12:38.527948 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.528075 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4186701}
I20240503 02:12:38.529431 5114 tablet_replica.cc:330] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:38.530535 5114 raft_consensus.cc:2229] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.531024 5114 raft_consensus.cc:2258] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:38.528153 4275 rpcz_store.cc:269] 0503 02:12:36.949687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.425481 (+475794us) service_pool.cc:225] Handling call
0503 02:12:37.719887 (+294406us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:38.526267 (+806380us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:38.290341 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:38.290486 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:38.290495 (+ 9us) write_op.cc:435] Acquired schema lock
0503 02:12:38.290500 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:38.290619 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:38.290633 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:38.290641 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:38.290718 (+ 77us) tablet.cc:699] Row locks acquired
0503 02:12:38.290723 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:38.290761 (+ 38us) write_op.cc:270] Start()
0503 02:12:38.290798 (+ 37us) write_op.cc:276] Timestamp: P: 1714702358290699 usec, L: 0
0503 02:12:38.290804 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:38.317825 (+ 27021us) log.cc:842] Serialized 527 byte log entry
0503 02:12:38.331708 (+ 13883us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 000000000095519a 00007fd3f2a03162 00007fd3f3d6be32 00007fd3f3d6a959 00007fd3f3d7286b 00007fd3f3d726e8 0000000000a60bcc 00007fd3e7204b74 00007fd3e720819b 00007fd3e7208018 0000000000a60bcc
0503 02:12:38.343314 (+ 11606us) op_driver.cc:444] REPLICATION: finished
0503 02:12:38.343580 (+ 266us) write_op.cc:301] APPLY: starting
0503 02:12:38.343595 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:38.343640 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:38.343643 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:38.343694 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:38.343730 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:38.343737 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:38.343806 (+ 69us) log.cc:842] Serialized 23 byte log entry
0503 02:12:38.343859 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.343874 (+ 15us) write_op.cc:454] Released schema lock
0503 02:12:38.343881 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:38.343933 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:38.343936 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":36,"spinlock_wait_cycles":674037632,"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":402,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":1449,"prepare.run_wall_time_us":41438,"raft.queue_time_us":844,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":3967,"replication_time_us":52484,"spinlock_wait_cycles":61621120}]]}
I20240503 02:12:38.536530 5114 ts_tablet_manager.cc:1875] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.539499 6374 raft_consensus.cc:482] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.539696 6374 raft_consensus.cc:504] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.540359 6374 leader_election.cc:288] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.540504 5106 ts_tablet_manager.cc:1398] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.263s user 0.002s sys 0.001s
I20240503 02:12:38.540534 4633 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 114) took 1429 ms. Trace:
I20240503 02:12:38.540624 4633 rpcz_store.cc:276] 0503 02:12:37.070337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.070822 (+ 485us) service_pool.cc:225] Handling call
0503 02:12:37.564812 (+493990us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x616000182e54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.052096 (+487284us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x6160000d3354. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.243866 (+191770us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x616000151654. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.415769 (+171903us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x616000181c54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.500296 (+ 84527us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":906968192}
I20240503 02:12:38.540814 5106 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:38.540519 4608 consensus_peers.cc:485] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22776c81d1a24d368abb68274cf32a0b. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:38.542268 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 72559.5us
I20240503 02:12:38.542501 5106 tablet_bootstrap.cc:654] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:38.543453 4608 leader_election.cc:341] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.543462 6382 consensus_queue.cc:1035] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.547163 5159 ts_tablet_manager.cc:1888] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.547981 5159 log.cc:1196] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/885129a4efc5475b97b7ac68e1db5e1d
I20240503 02:12:38.550837 6427 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:38.553288 5159 ts_tablet_manager.cc:1909] T 885129a4efc5475b97b7ac68e1db5e1d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.555337 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b21a3010a4470288fbdd684ef64e77" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.555953 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet da12dd28e03f462b905d1844e2ad3340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:38.556715 4608 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:38.559409 4608 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:38.558215 6411 tablet.cc:1717] T 001fe4bb80c24d8e97d971a7130b2296 P 1f50719d94564844b15e1a9211d48a81: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:38.559551 4608 leader_election.cc:302] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.560710 6374 raft_consensus.cc:2726] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:38.561427 4608 consensus_peers.cc:485] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:38.562346 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3f18eaa565c84fe18b01cefd0d501ad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.562582 6407 tablet.cc:1717] T 001fe4bb80c24d8e97d971a7130b2296 P f4f9f079230a468f90e5e3a58f254b91: Skipping requested alter to schema version 2, tablet already version 2
W20240503 02:12:38.563005 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f18eaa565c84fe18b01cefd0d501ad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.564522 6400 tablet.cc:1717] T 001fe4bb80c24d8e97d971a7130b2296 P acc6d33969e74c209310005ae911c405: Skipping requested alter to schema version 2, tablet already version 2
I20240503 02:12:38.571933 5868 raft_consensus.cc:482] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.572168 5868 raft_consensus.cc:504] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.572768 5868 leader_election.cc:288] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.573622 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.574774 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91abaf51abb4c2ca65412c3771c1ef2" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:38.575037 4811 raft_consensus.cc:2379] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:38.575457 4462 leader_election.cc:302] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.575870 5868 raft_consensus.cc:3037] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.578670 5868 raft_consensus.cc:2726] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:38.579284 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 78b21a3010a4470288fbdd684ef64e77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78b21a3010a4470288fbdd684ef64e77 already in progress: creating tablet
W20240503 02:12:38.579636 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 211dfd4049684f5e8af927b2a19263b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211dfd4049684f5e8af927b2a19263b7 already in progress: creating tablet
W20240503 02:12:38.579883 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 50e04292742646049f39b6aa29338d75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e04292742646049f39b6aa29338d75 already in progress: creating tablet
W20240503 02:12:38.580129 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6e635f283288406aa16b3941be64e777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e635f283288406aa16b3941be64e777 already in progress: creating tablet
I20240503 02:12:38.581331 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3e2c17a4387e47aa99c76caca5b18154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:38.581935 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:38.582033 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e2c17a4387e47aa99c76caca5b18154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.582376 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.582724 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.583041 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet d271d2c112ba44b68d2933a6ebfe7a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.583406 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.583763 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 with delete_type TABLET_DATA_DELETED (replaced by tablet 04d555c8d8844ee49c960a5279485480) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.585489 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:20 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.585809 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.586148 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.586465 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.586802 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:38.587693 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 16b3d1e7ce83425db0bd56457d9f2544 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:38.588337 6353 raft_consensus.cc:482] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.588483 6353 raft_consensus.cc:504] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:38.589124 6353 leader_election.cc:288] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.591256 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.596093 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:38.596516 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:38.597923 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 655) took 2519 ms. Trace:
W20240503 02:12:38.599195 4746 leader_election.cc:341] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.599469 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:38.599967 4746 leader_election.cc:341] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.600059 4746 leader_election.cc:302] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.600488 6353 raft_consensus.cc:2726] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.596905 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:38.601358 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]). Expected version 1 got 0
I20240503 02:12:38.601552 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:38.601752 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7d919b9262ac48cbb8b3f893918b5590 (table default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95]). Expected version 3 got 0
I20240503 02:12:38.601773 5159 rpcz_store.cc:276] 0503 02:12:36.078033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.078076 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:38.413294 (+2335218us) ts_tablet_manager.cc:1120] Deleting tablet 885129a4efc5475b97b7ac68e1db5e1d
0503 02:12:38.413310 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.500698 (+ 87388us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.546506 (+ 45808us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.597807 (+ 51301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2098679}
I20240503 02:12:38.603869 5159 tablet_replica.cc:330] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.605077 5105 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.605072 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:38.605216 5105 ts_tablet_manager.cc:1367] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.196s user 0.003s sys 0.001s
I20240503 02:12:38.605314 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:38.605341 5159 raft_consensus.cc:2229] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.609395 5105 raft_consensus.cc:348] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.609737 5105 raft_consensus.cc:374] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.609860 5105 raft_consensus.cc:729] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.610813 5159 raft_consensus.cc:2258] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.610941 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:38.611188 4274 catalog_manager.cc:5484] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 2 to 3, leader changed from 1f50719d94564844b15e1a9211d48a81 (127.4.17.66) to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 3 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.611145 5105 consensus_queue.cc:260] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.611840 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]). Expected version 1 got 0
I20240503 02:12:38.612563 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
W20240503 02:12:38.617437 4268 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 101) took 1170 ms (client timeout 1000 ms). Trace:
W20240503 02:12:38.617596 4268 rpcz_store.cc:269] 0503 02:12:37.446473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.457473 (+ 11000us) service_pool.cc:225] Handling call
0503 02:12:37.721462 (+263989us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:38.617370 (+895908us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":37,"spinlock_wait_cycles":603999104}
I20240503 02:12:38.621738 6374 consensus_queue.cc:1035] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.626497 4496 tablet_service.cc:1467] Processing CreateTablet for tablet be423b7d85d14bbbae0000648a94c183 (DEFAULT_TABLE table=default.Table_42acc9ebe0264f3a833dcf672cf1b830 [id=3de81c7cd7c74fd9a68661fd0d4d2381]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.626927 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423b7d85d14bbbae0000648a94c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.627188 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 1b146a785e814d7298f423168de239f6 (DEFAULT_TABLE table=default.Table_78930698028a4f4ab89c3beaf915f453 [id=5c2d659f89e94bb7a8f055e3587c1258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.627528 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b146a785e814d7298f423168de239f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.634956 5103 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.635195 5103 ts_tablet_manager.cc:1367] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.164s user 0.004s sys 0.002s
I20240503 02:12:38.637526 5103 raft_consensus.cc:348] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.637790 5103 raft_consensus.cc:374] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.637881 5103 raft_consensus.cc:729] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.638136 5103 consensus_queue.cc:260] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.625079 5159 ts_tablet_manager.cc:1875] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.625161 5105 ts_tablet_manager.cc:1398] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:12:38.626524 4783 tablet_service.cc:1467] Processing CreateTablet for tablet be423b7d85d14bbbae0000648a94c183 (DEFAULT_TABLE table=default.Table_42acc9ebe0264f3a833dcf672cf1b830 [id=3de81c7cd7c74fd9a68661fd0d4d2381]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.643497 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.636385 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 1b146a785e814d7298f423168de239f6 (DEFAULT_TABLE table=default.Table_78930698028a4f4ab89c3beaf915f453 [id=5c2d659f89e94bb7a8f055e3587c1258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.637389 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 0794df2bfc5e45db95cf4c745892e695 (DEFAULT_TABLE table=default.Table_07f00cb83ef24520a8b4e944ab24b09f [id=796958c1ea26451ea6c8f64243387885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.638150 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdfaff4b319415db858f06c47829933 (DEFAULT_TABLE table=default.Table_5c3f0c84a0db4405ad8fed6849923e96 [id=51bb4e5b438f4d66bbc460b49901bd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.638402 6296 consensus_queue.cc:1035] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.639452 4785 tablet_service.cc:1467] Processing CreateTablet for tablet a3374e095846463a859bb642c2f42167 (DEFAULT_TABLE table=default.Table_e800e34461a64783bd120662f37d31a3 [id=65cae3efee7646d4b9676a9fca01910a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.640712 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbbfa2d1a49400c809d5b476036cdcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.641276 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b21a3010a4470288fbdd684ef64e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.642032 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4dac17dca3442d8934ac357779b78a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.642442 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.642829 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.642829 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.645473 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.645561 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2b3293557cf140d9a3edf888adb2f459"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:38.646186 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b3293557cf140d9a3edf888adb2f459 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:38.638785 4493 tablet_service.cc:1467] Processing CreateTablet for tablet a3374e095846463a859bb642c2f42167 (DEFAULT_TABLE table=default.Table_e800e34461a64783bd120662f37d31a3 [id=65cae3efee7646d4b9676a9fca01910a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.649264 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3374e095846463a859bb642c2f42167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.612792 4274 catalog_manager.cc:5484] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.651728 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423b7d85d14bbbae0000648a94c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.651926 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3374e095846463a859bb642c2f42167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.652102 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0794df2bfc5e45db95cf4c745892e695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.640239 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb6f5087475411ea280a72ec948ad31 (DEFAULT_TABLE table=default.Table_c77b55b8023b459c96d97e8d9c8f7986 [id=4b238c1a9bcb4638844c6aa478082c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.641518 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdfaff4b319415db858f06c47829933 (DEFAULT_TABLE table=default.Table_5c3f0c84a0db4405ad8fed6849923e96 [id=51bb4e5b438f4d66bbc460b49901bd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.653061 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb6f5087475411ea280a72ec948ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.653270 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdfaff4b319415db858f06c47829933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.657228 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b146a785e814d7298f423168de239f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.641534 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 0794df2bfc5e45db95cf4c745892e695 (DEFAULT_TABLE table=default.Table_07f00cb83ef24520a8b4e944ab24b09f [id=796958c1ea26451ea6c8f64243387885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.659654 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0794df2bfc5e45db95cf4c745892e695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.660837 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdfaff4b319415db858f06c47829933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.665222 5105 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.668447 5103 ts_tablet_manager.cc:1398] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20240503 02:12:38.678776 5105 tablet_bootstrap.cc:654] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.679795 5159 ts_tablet_manager.cc:1888] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.679981 5159 log.cc:1196] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/159ea1eb32394b04b72258d24196202b
I20240503 02:12:38.680555 5159 ts_tablet_manager.cc:1909] T 159ea1eb32394b04b72258d24196202b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.681814 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 653) took 2624 ms. Trace:
I20240503 02:12:38.681991 5159 rpcz_store.cc:276] 0503 02:12:36.057382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.077685 (+ 20303us) service_pool.cc:225] Handling call
0503 02:12:38.603580 (+2525895us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:38.603592 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.674689 (+ 71097us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.679634 (+ 4945us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.681795 (+ 2161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2958,"tablet-delete.queue_time_us":2288606}
I20240503 02:12:38.682705 5159 tablet_replica.cc:330] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.683001 5159 raft_consensus.cc:2229] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.683465 5159 raft_consensus.cc:2258] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.685467 5159 ts_tablet_manager.cc:1875] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.686419 5105 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.686754 5105 ts_tablet_manager.cc:1367] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 02:12:38.688520 5268 master_runner.cc:186] Error getting master registration for 127.4.17.124:46689: OK, instance_id { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" instance_seqno: 1714702356513402 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:12:38.688876 5268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:12:38.689414 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:38.689764 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]). Expected version 1 got 0
I20240503 02:12:38.690003 4274 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:38.690261 5105 raft_consensus.cc:348] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.690490 5105 raft_consensus.cc:374] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.690562 5105 raft_consensus.cc:729] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.690815 5105 consensus_queue.cc:260] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.690858 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_081e142b6f9e45b5b4cb3987b8f31ce1" } new_table_name: "default.Table_c8f443440ac54575a543e777004dca5a" modify_external_catalogs: true
I20240503 02:12:38.693166 4518 raft_consensus.cc:1261] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:12:38.694016 6057 consensus_queue.cc:1035] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:38.695712 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 885129a4efc5475b97b7ac68e1db5e1d (table default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]) successfully deleted
I20240503 02:12:38.696722 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 159ea1eb32394b04b72258d24196202b (table default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]) successfully deleted
I20240503 02:12:38.698669 4495 tablet_service.cc:1467] Processing CreateTablet for tablet e0ca67743c904927a2ad6c8ca099fc1c (DEFAULT_TABLE table=default.Table_7b6f1b8dd48144eb88d5710246516d40 [id=1741caf9fbfa42a69bec665950feea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.699127 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ca67743c904927a2ad6c8ca099fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.699247 4644 tablet_service.cc:1467] Processing CreateTablet for tablet be423b7d85d14bbbae0000648a94c183 (DEFAULT_TABLE table=default.Table_42acc9ebe0264f3a833dcf672cf1b830 [id=3de81c7cd7c74fd9a68661fd0d4d2381]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.699631 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be423b7d85d14bbbae0000648a94c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.701653 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 17afd88bf6c040ad9e8d571eccd348dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.701754 6407 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.701861 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:38.702009 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb6f5087475411ea280a72ec948ad31 (DEFAULT_TABLE table=default.Table_c77b55b8023b459c96d97e8d9c8f7986 [id=4b238c1a9bcb4638844c6aa478082c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.702548 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb6f5087475411ea280a72ec948ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.704744 5103 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.706471 6400 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.713557 5114 ts_tablet_manager.cc:1888] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.716095 6400 tablet.cc:1722] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:38.724376 5159 ts_tablet_manager.cc:1888] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.724939 5159 log.cc:1196] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b161628065704c798020edbe098844c2
I20240503 02:12:38.725893 5105 ts_tablet_manager.cc:1398] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:12:38.726363 5159 ts_tablet_manager.cc:1909] T b161628065704c798020edbe098844c2 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.726554 4673 raft_consensus.cc:1261] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Refusing update from remote peer acc6d33969e74c209310005ae911c405: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20240503 02:12:38.726660 4673 pending_rounds.cc:76] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Aborting all ops after (but not including) 2
I20240503 02:12:38.726699 4673 pending_rounds.cc:97] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20240503 02:12:38.726786 4673 raft_consensus.cc:2866] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:12:38.727711 5114 log.cc:1196] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/5f56701ed6424c60bf1a985eea1936fe
I20240503 02:12:38.728120 5114 ts_tablet_manager.cc:1909] T 5f56701ed6424c60bf1a985eea1936fe P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:38.728331 6353 consensus_queue.cc:1035] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:38.718147 5103 tablet_bootstrap.cc:654] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.730798 6427 tablet.cc:1722] T b14241dd76864e8a93d0279d3ab8f3ac P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:38.732654 5103 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.733026 5105 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.733489 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 654) took 2655 ms. Trace:
I20240503 02:12:38.733675 5159 rpcz_store.cc:276] 0503 02:12:36.077903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.077950 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:38.682615 (+2604665us) ts_tablet_manager.cc:1120] Deleting tablet b161628065704c798020edbe098844c2
0503 02:12:38.682649 (+ 34us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.709161 (+ 26512us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.724097 (+ 14936us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.733469 (+ 9372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359353}
I20240503 02:12:38.734452 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 656) took 2356 ms. Trace:
I20240503 02:12:38.734644 5159 rpcz_store.cc:276] 0503 02:12:36.377686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.377924 (+ 238us) service_pool.cc:225] Handling call
0503 02:12:38.734294 (+2356370us) ts_tablet_manager.cc:1120] Deleting tablet b080fc5ffc564420a0b7f2b7460ebbbb
0503 02:12:38.734305 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.734435 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2356085}
I20240503 02:12:38.735306 5159 tablet_replica.cc:330] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.735944 5159 raft_consensus.cc:2229] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.736339 5159 raft_consensus.cc:2258] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.737504 5159 ts_tablet_manager.cc:1875] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.737643 5103 ts_tablet_manager.cc:1367] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240503 02:12:38.737270 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 608) took 4394 ms. Trace:
I20240503 02:12:38.740043 5114 rpcz_store.cc:276] 0503 02:12:34.342585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.342726 (+ 141us) service_pool.cc:225] Handling call
0503 02:12:38.529385 (+4186659us) ts_tablet_manager.cc:1120] Deleting tablet 5f56701ed6424c60bf1a985eea1936fe
0503 02:12:38.529394 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.640973 (+111579us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.713476 (+ 72503us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.737245 (+ 23769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4186351}
I20240503 02:12:38.741108 5105 tablet_bootstrap.cc:654] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.743337 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 0794df2bfc5e45db95cf4c745892e695 (DEFAULT_TABLE table=default.Table_07f00cb83ef24520a8b4e944ab24b09f [id=796958c1ea26451ea6c8f64243387885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.744104 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdfaff4b319415db858f06c47829933 (DEFAULT_TABLE table=default.Table_5c3f0c84a0db4405ad8fed6849923e96 [id=51bb4e5b438f4d66bbc460b49901bd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.744482 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdfaff4b319415db858f06c47829933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.745455 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 609) took 4174 ms. Trace:
I20240503 02:12:38.747938 5114 rpcz_store.cc:276] 0503 02:12:34.571173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.571230 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:38.745311 (+4174081us) ts_tablet_manager.cc:1120] Deleting tablet f5fd27023b3f47f281dfb5474ea3a4c6
0503 02:12:38.745322 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.745437 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4169373}
I20240503 02:12:38.751096 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 1b146a785e814d7298f423168de239f6 (DEFAULT_TABLE table=default.Table_78930698028a4f4ab89c3beaf915f453 [id=5c2d659f89e94bb7a8f055e3587c1258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.748347 5103 raft_consensus.cc:348] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.751787 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0794df2bfc5e45db95cf4c745892e695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.753346 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b146a785e814d7298f423168de239f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.753340 4636 tablet_service.cc:1467] Processing CreateTablet for tablet a3374e095846463a859bb642c2f42167 (DEFAULT_TABLE table=default.Table_e800e34461a64783bd120662f37d31a3 [id=65cae3efee7646d4b9676a9fca01910a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.751437 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:38.751579 5103 raft_consensus.cc:374] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.754153 5159 ts_tablet_manager.cc:1888] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.754424 5103 raft_consensus.cc:729] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.754379 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb6f5087475411ea280a72ec948ad31 (DEFAULT_TABLE table=default.Table_c77b55b8023b459c96d97e8d9c8f7986 [id=4b238c1a9bcb4638844c6aa478082c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.754460 5159 log.cc:1196] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/4ced4dfb2cf541af98d50d22d1a0f0ac
I20240503 02:12:38.754691 5103 consensus_queue.cc:260] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.755259 5103 ts_tablet_manager.cc:1398] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240503 02:12:38.751808 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 622) took 3979 ms. Trace:
I20240503 02:12:38.755527 5114 rpcz_store.cc:276] 0503 02:12:34.769833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.769896 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:38.748981 (+3979085us) ts_tablet_manager.cc:1120] Deleting tablet 7447242714c844e685faca5aed71a7f7
0503 02:12:38.748992 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.749098 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3978840}
I20240503 02:12:38.755756 5159 ts_tablet_manager.cc:1909] T 4ced4dfb2cf541af98d50d22d1a0f0ac P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.756072 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 617) took 4011 ms. Trace:
I20240503 02:12:38.756453 5114 rpcz_store.cc:276] 0503 02:12:34.744711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.745096 (+ 385us) service_pool.cc:225] Handling call
0503 02:12:38.755949 (+4010853us) ts_tablet_manager.cc:1120] Deleting tablet 84f6279f16dc42d69e794e3c6586163d
0503 02:12:38.755958 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.756057 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3945185}
I20240503 02:12:38.756744 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 657) took 2378 ms. Trace:
I20240503 02:12:38.757185 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 614) took 4012 ms. Trace:
I20240503 02:12:38.757560 5114 rpcz_store.cc:276] 0503 02:12:34.744342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.745984 (+ 1642us) service_pool.cc:225] Handling call
0503 02:12:38.757054 (+4011070us) ts_tablet_manager.cc:1120] Deleting tablet b8474c841cb64373a223ad76b43ecf5f
0503 02:12:38.757065 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.757157 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3946213}
I20240503 02:12:38.758941 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 615) took 4014 ms. Trace:
I20240503 02:12:38.759589 5114 rpcz_store.cc:276] 0503 02:12:34.744495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746331 (+ 1836us) service_pool.cc:225] Handling call
0503 02:12:38.758815 (+4012484us) ts_tablet_manager.cc:1120] Deleting tablet a8ce4b33f90c40c48cf0a538a1a7fde4
0503 02:12:38.758824 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.758926 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3947931}
I20240503 02:12:38.761345 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3374e095846463a859bb642c2f42167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.761528 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb6f5087475411ea280a72ec948ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.765106 5114 tablet_replica.cc:330] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:38.765571 5114 raft_consensus.cc:2229] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.765998 5114 raft_consensus.cc:2258] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.768890 6183 raft_consensus.cc:482] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.770087 6183 raft_consensus.cc:504] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.770524 5114 ts_tablet_manager.cc:1875] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.770741 6183 leader_election.cc:288] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.772522 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.771299 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b962c9264f87b798df33544cb15d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:38.775635 4811 raft_consensus.cc:2379] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:38.771624 5105 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.776156 5105 ts_tablet_manager.cc:1367] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240503 02:12:38.776474 4462 leader_election.cc:302] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.776878 6183 raft_consensus.cc:3037] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.777540 5105 raft_consensus.cc:348] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.777813 5105 raft_consensus.cc:374] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.777915 5105 raft_consensus.cc:729] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.778240 5105 consensus_queue.cc:260] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.779089 5105 ts_tablet_manager.cc:1398] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:38.779465 6183 raft_consensus.cc:2726] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.779551 5105 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.780825 5159 rpcz_store.cc:276] 0503 02:12:36.378346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.382276 (+ 3930us) service_pool.cc:225] Handling call
0503 02:12:38.735191 (+2352915us) ts_tablet_manager.cc:1120] Deleting tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
0503 02:12:38.735225 (+ 34us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.747467 (+ 12242us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.753041 (+ 5574us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.756691 (+ 3650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352693}
I20240503 02:12:38.782449 5159 tablet_replica.cc:330] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.783242 5159 raft_consensus.cc:2229] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.783829 5159 raft_consensus.cc:2258] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.784801 5105 tablet_bootstrap.cc:654] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.785486 6374 consensus_queue.cc:1035] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.786413 5159 ts_tablet_manager.cc:1875] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.798424 6183 consensus_queue.cc:1035] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.816208 5159 ts_tablet_manager.cc:1888] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.816668 5114 ts_tablet_manager.cc:1888] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.817854 5114 log.cc:1196] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/396b48865e954d05b9ba8084a2e7f29b
I20240503 02:12:38.818356 5114 ts_tablet_manager.cc:1909] T 396b48865e954d05b9ba8084a2e7f29b P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:38.819312 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 616) took 4074 ms. Trace:
I20240503 02:12:38.819526 5159 log.cc:1196] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3b72c20a4b5a4d2ea266f031733d1aab
I20240503 02:12:38.820039 5159 ts_tablet_manager.cc:1909] T 3b72c20a4b5a4d2ea266f031733d1aab P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.822057 5114 rpcz_store.cc:276] 0503 02:12:34.744625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746535 (+ 1910us) service_pool.cc:225] Handling call
0503 02:12:38.765051 (+4018516us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:38.765063 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.793907 (+ 28844us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.816530 (+ 22623us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.819285 (+ 2755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3954121}
I20240503 02:12:38.823993 6183 consensus_queue.cc:1035] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.828841 5105 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.829071 5105 ts_tablet_manager.cc:1367] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.025s
I20240503 02:12:38.829908 5105 raft_consensus.cc:348] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.830130 5105 raft_consensus.cc:374] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.830303 5105 raft_consensus.cc:729] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.830756 5105 consensus_queue.cc:260] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.831367 5105 ts_tablet_manager.cc:1398] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:38.831902 6353 raft_consensus.cc:482] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.832140 6353 raft_consensus.cc:504] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.831959 5105 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.833065 6353 leader_election.cc:288] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.833916 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.834226 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 658) took 2451 ms. Trace:
W20240503 02:12:38.834450 4746 leader_election.cc:341] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:38.836134 4746 consensus_peers.cc:485] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f56701ed6424c60bf1a985eea1936fe. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:38.836946 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:38.837559 4746 leader_election.cc:341] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.837715 4746 leader_election.cc:302] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.838133 6353 raft_consensus.cc:2726] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.839062 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 618) took 4090 ms. Trace:
I20240503 02:12:38.839533 5159 rpcz_store.cc:276] 0503 02:12:36.382659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.382702 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:38.782306 (+2399604us) ts_tablet_manager.cc:1120] Deleting tablet 3b72c20a4b5a4d2ea266f031733d1aab
0503 02:12:38.782317 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.790383 (+ 8066us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.816050 (+ 25667us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.834204 (+ 18154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399388}
I20240503 02:12:38.839280 5114 rpcz_store.cc:276] 0503 02:12:34.744841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746669 (+ 1828us) service_pool.cc:225] Handling call
0503 02:12:38.834762 (+4088093us) ts_tablet_manager.cc:1120] Deleting tablet 04d555c8d8844ee49c960a5279485480
0503 02:12:38.834774 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.834880 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4022850}
I20240503 02:12:38.842067 6349 raft_consensus.cc:482] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.842255 6349 raft_consensus.cc:504] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.842942 6349 leader_election.cc:288] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.843819 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:38.844412 4746 leader_election.cc:341] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.847707 5105 tablet_bootstrap.cc:654] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.850158 5159 tablet_replica.cc:330] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.850399 5159 raft_consensus.cc:2229] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.850742 5159 raft_consensus.cc:2258] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.851464 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:38.851295 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 619) took 4106 ms. Trace:
I20240503 02:12:38.851769 5114 rpcz_store.cc:276] 0503 02:12:34.745007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.746795 (+ 1788us) service_pool.cc:225] Handling call
0503 02:12:38.840443 (+4093648us) ts_tablet_manager.cc:1120] Deleting tablet 8135907b78284d7ca74396cc77dd0612
0503 02:12:38.851111 (+ 10668us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.851264 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4029411}
W20240503 02:12:38.852205 4746 leader_election.cc:341] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.852404 4746 leader_election.cc:302] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:38.852712 4632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 117) took 1672 ms. Trace:
I20240503 02:12:38.852854 6353 raft_consensus.cc:2726] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.853588 5114 tablet_replica.cc:330] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:38.853652 4632 rpcz_store.cc:276] 0503 02:12:37.180213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.180281 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:38.211023 (+1030742us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x6160001f1554. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.444875 (+233852us) spinlock_profiling.cc:243] Waited 91.8 ms on lock 0x616000026554. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.602950 (+158075us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x6160000b4454. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.841079 (+238129us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x6160000b5054. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.852682 (+ 11603us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1350923008}
I20240503 02:12:38.854517 5114 raft_consensus.cc:2229] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.853545 4631 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 116) took 1770 ms. Trace:
I20240503 02:12:38.855378 4631 rpcz_store.cc:276] 0503 02:12:37.083286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.083333 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:38.082324 (+998991us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x616000151c54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.444365 (+362041us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x616000026554. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.599791 (+155426us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x6160000b4454. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.837669 (+237878us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x6160000b5054. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.853504 (+ 15835us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1910440320}
I20240503 02:12:38.855521 5159 ts_tablet_manager.cc:1875] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.857899 5106 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:38.858224 5106 ts_tablet_manager.cc:1367] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.317s user 0.011s sys 0.005s
I20240503 02:12:38.859282 5106 raft_consensus.cc:348] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.859556 5106 raft_consensus.cc:374] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.859694 5106 raft_consensus.cc:729] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.859993 5106 consensus_queue.cc:260] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.860633 5106 ts_tablet_manager.cc:1398] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:38.861234 5106 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:38.863329 6183 raft_consensus.cc:482] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.863504 6183 raft_consensus.cc:504] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:38.864830 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8c27e37ebd0f49cdba7653548acbe1ea consensus state has no leader
I20240503 02:12:38.865317 6183 leader_election.cc:288] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:38.866487 4462 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:38.877310 6183 raft_consensus.cc:482] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.877707 6183 raft_consensus.cc:504] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.879318 5159 ts_tablet_manager.cc:1888] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.879482 5159 log.cc:1196] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/53017352b8f34955b116bbfd32f04854
I20240503 02:12:38.879320 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:38.879681 4808 raft_consensus.cc:2379] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:38.879788 6183 leader_election.cc:288] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:38.879930 5159 ts_tablet_manager.cc:1909] T 53017352b8f34955b116bbfd32f04854 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.880242 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 659) took 2497 ms. Trace:
I20240503 02:12:38.880328 5159 rpcz_store.cc:276] 0503 02:12:36.383005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.384187 (+ 1182us) service_pool.cc:225] Handling call
0503 02:12:38.850073 (+2465886us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:38.850086 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.876189 (+ 26103us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.879206 (+ 3017us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.880222 (+ 1016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359904}
I20240503 02:12:38.882992 4783 tablet_service.cc:1467] Processing CreateTablet for tablet e0ca67743c904927a2ad6c8ca099fc1c (DEFAULT_TABLE table=default.Table_7b6f1b8dd48144eb88d5710246516d40 [id=1741caf9fbfa42a69bec665950feea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.883905 5159 tablet_replica.cc:330] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.883884 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ca67743c904927a2ad6c8ca099fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.884150 5159 raft_consensus.cc:2229] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:38.884500 5159 raft_consensus.cc:2258] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.889147 5159 ts_tablet_manager.cc:1875] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.895300 5159 ts_tablet_manager.cc:1888] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.895433 5159 log.cc:1196] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5f56701ed6424c60bf1a985eea1936fe
I20240503 02:12:38.895834 5159 ts_tablet_manager.cc:1909] T 5f56701ed6424c60bf1a985eea1936fe P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.896196 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 660) took 2508 ms. Trace:
I20240503 02:12:38.898213 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 5f56701ed6424c60bf1a985eea1936fe (table default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]) successfully deleted
W20240503 02:12:38.898926 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f5fd27023b3f47f281dfb5474ea3a4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5fd27023b3f47f281dfb5474ea3a4c6 already in progress: creating tablet
W20240503 02:12:38.899333 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7447242714c844e685faca5aed71a7f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7447242714c844e685faca5aed71a7f7 already in progress: creating tablet
W20240503 02:12:38.899874 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b8474c841cb64373a223ad76b43ecf5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8474c841cb64373a223ad76b43ecf5f already in progress: creating tablet
W20240503 02:12:38.900218 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ce4b33f90c40c48cf0a538a1a7fde4 already in progress: creating tablet
I20240503 02:12:38.900557 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 396b48865e954d05b9ba8084a2e7f29b (table default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]) successfully deleted
W20240503 02:12:38.900961 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 04d555c8d8844ee49c960a5279485480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04d555c8d8844ee49c960a5279485480 already in progress: creating tablet
I20240503 02:12:38.901069 5105 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:38.896275 5159 rpcz_store.cc:276] 0503 02:12:36.388024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.388901 (+ 877us) service_pool.cc:225] Handling call
0503 02:12:38.883857 (+2494956us) ts_tablet_manager.cc:1120] Deleting tablet 5f56701ed6424c60bf1a985eea1936fe
0503 02:12:38.883867 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.892203 (+ 8336us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.895182 (+ 2979us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.896159 (+ 977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387537}
I20240503 02:12:38.913203 5105 ts_tablet_manager.cc:1367] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.002s
I20240503 02:12:38.913981 5105 raft_consensus.cc:348] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.914211 5105 raft_consensus.cc:374] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.914264 5105 raft_consensus.cc:729] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.914469 5105 consensus_queue.cc:260] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:38.914088 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 682) took 1961 ms. Trace:
I20240503 02:12:38.914958 5105 ts_tablet_manager.cc:1398] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:38.914932 5159 rpcz_store.cc:276] 0503 02:12:36.952364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.952443 (+ 79us) service_pool.cc:225] Handling call
0503 02:12:38.913906 (+1961463us) ts_tablet_manager.cc:1120] Deleting tablet 5cd296bb3b574d6cbb0252ad10ada00c
0503 02:12:38.913919 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.914069 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961243}
I20240503 02:12:38.915236 5105 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:38.915452 5159 tablet_replica.cc:330] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:38.915637 5159 raft_consensus.cc:2229] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.915796 5159 raft_consensus.cc:2258] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.917141 5105 tablet_bootstrap.cc:654] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.921010 5159 ts_tablet_manager.cc:1875] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.927796 5114 raft_consensus.cc:2258] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.928556 5114 ts_tablet_manager.cc:1875] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.932102 5159 ts_tablet_manager.cc:1888] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:38.943197 5114 ts_tablet_manager.cc:1888] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:38.943415 5114 log.cc:1196] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/2f053537029a44d5a519b612d09d19a1
I20240503 02:12:38.943876 5114 ts_tablet_manager.cc:1909] T 2f053537029a44d5a519b612d09d19a1 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:38.944744 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 620) took 4188 ms. Trace:
I20240503 02:12:38.944883 5114 rpcz_store.cc:276] 0503 02:12:34.756122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.756196 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:38.853525 (+4097329us) ts_tablet_manager.cc:1120] Deleting tablet 2f053537029a44d5a519b612d09d19a1
0503 02:12:38.853539 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.939722 (+ 86183us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.943064 (+ 3342us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.944720 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4042304}
I20240503 02:12:38.945379 5114 tablet_replica.cc:330] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:38.945593 5114 raft_consensus.cc:2229] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:38.945789 5114 raft_consensus.cc:2258] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:38.949054 5114 ts_tablet_manager.cc:1875] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:38.951247 5159 log.cc:1196] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7178760c21904207b6c8317093871f63
I20240503 02:12:38.951673 5159 ts_tablet_manager.cc:1909] T 7178760c21904207b6c8317093871f63 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:38.953786 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 681) took 2001 ms. Trace:
I20240503 02:12:38.954002 5159 rpcz_store.cc:276] 0503 02:12:36.952278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.952574 (+ 296us) service_pool.cc:225] Handling call
0503 02:12:38.915397 (+1962823us) ts_tablet_manager.cc:1120] Deleting tablet 7178760c21904207b6c8317093871f63
0503 02:12:38.915408 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.924430 (+ 9022us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.931970 (+ 7540us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:38.953759 (+ 21789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1962644}
I20240503 02:12:38.955173 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 683) took 2002 ms. Trace:
I20240503 02:12:38.955355 5159 rpcz_store.cc:276] 0503 02:12:36.952442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.952709 (+ 267us) service_pool.cc:225] Handling call
0503 02:12:38.954894 (+2002185us) ts_tablet_manager.cc:1120] Deleting tablet 6d08e7d17f1644749a9a2e7175b3191a
0503 02:12:38.954906 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.955118 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002019}
I20240503 02:12:38.956146 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 684) took 2003 ms. Trace:
I20240503 02:12:38.956293 5159 rpcz_store.cc:276] 0503 02:12:36.952528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.952876 (+ 348us) service_pool.cc:225] Handling call
0503 02:12:38.955909 (+2003033us) ts_tablet_manager.cc:1120] Deleting tablet 0a8c57ddf3d8428e8fa5ffb061dc054b
0503 02:12:38.955941 (+ 32us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.956099 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002874}
I20240503 02:12:38.957442 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 701) took 1125 ms. Trace:
I20240503 02:12:38.957594 5159 rpcz_store.cc:276] 0503 02:12:37.831832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.832653 (+ 821us) service_pool.cc:225] Handling call
0503 02:12:38.956809 (+1124156us) ts_tablet_manager.cc:1120] Deleting tablet 6b15997a94b74ad3808d965acfb5d155
0503 02:12:38.956819 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.957401 (+ 582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123940}
I20240503 02:12:38.962177 5103 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.963711 5103 tablet_bootstrap.cc:654] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.865092 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65774b69a8bd4664a6b55b84eba37f7f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:38.974133 4462 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.974284 4462 leader_election.cc:302] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.881366 4651 tablet_service.cc:1467] Processing CreateTablet for tablet e0ca67743c904927a2ad6c8ca099fc1c (DEFAULT_TABLE table=default.Table_7b6f1b8dd48144eb88d5710246516d40 [id=1741caf9fbfa42a69bec665950feea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.893532 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394c9d374404fb199ab6ff170d7febe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:38.971876 6262 raft_consensus.cc:482] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:38.975201 5106 tablet_bootstrap.cc:654] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.976022 4462 leader_election.cc:302] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:38.976612 5868 raft_consensus.cc:2726] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.976949 5868 raft_consensus.cc:3037] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.976812 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_94e778467d034214b379477ebfe9694e" } modify_external_catalogs: true
I20240503 02:12:38.977226 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_94e778467d034214b379477ebfe9694e" } modify_external_catalogs: true
I20240503 02:12:38.978070 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_308844f35cb448378721f1da35d38c48" } new_table_name: "default.Table_2c9a62a49c0b48ef86699496645d06df" modify_external_catalogs: true
W20240503 02:12:38.981662 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8135907b78284d7ca74396cc77dd0612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8135907b78284d7ca74396cc77dd0612 already in progress: creating tablet
I20240503 02:12:38.984383 4775 tablet_service.cc:1467] Processing CreateTablet for tablet f11a1f9872714a3a91b1179e0d44a14a (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.984685 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 41531ddcbf66465094d8f06e1116b3be (DEFAULT_TABLE table=default.Table_cf5d681047e040968c0436492ac29ac6 [id=04018dec3d824068ae653291a488118b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.984774 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a1f9872714a3a91b1179e0d44a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.984798 5103 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:38.984905 5103 ts_tablet_manager.cc:1367] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.007s
I20240503 02:12:38.985071 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41531ddcbf66465094d8f06e1116b3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.985925 5103 raft_consensus.cc:348] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.986095 5103 raft_consensus.cc:374] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.986135 5103 raft_consensus.cc:729] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.986347 5103 consensus_queue.cc:260] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:38.987118 5103 ts_tablet_manager.cc:1398] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:38.987362 5103 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:38.989038 5103 tablet_bootstrap.cc:654] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.993510 4495 tablet_service.cc:1467] Processing CreateTablet for tablet f11a1f9872714a3a91b1179e0d44a14a (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.994246 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a1f9872714a3a91b1179e0d44a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.007983 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 41531ddcbf66465094d8f06e1116b3be (DEFAULT_TABLE table=default.Table_cf5d681047e040968c0436492ac29ac6 [id=04018dec3d824068ae653291a488118b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.008447 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41531ddcbf66465094d8f06e1116b3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.013785 4494 tablet_service.cc:1467] Processing CreateTablet for tablet a070840b5b734edb8508900af51a4de1 (DEFAULT_TABLE table=default.Table_f8cf5f90991c472a9fb961b4e6d6829e [id=ed34a88be4634b46aef3782d2e0300f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.014226 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070840b5b734edb8508900af51a4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.994196 5868 raft_consensus.cc:2726] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.025521 4788 tablet_service.cc:1467] Processing CreateTablet for tablet a070840b5b734edb8508900af51a4de1 (DEFAULT_TABLE table=default.Table_f8cf5f90991c472a9fb961b4e6d6829e [id=ed34a88be4634b46aef3782d2e0300f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.025969 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070840b5b734edb8508900af51a4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.033517 5105 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.034018 5105 ts_tablet_manager.cc:1367] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.037s
I20240503 02:12:38.975080 6262 raft_consensus.cc:504] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.034766 6262 leader_election.cc:288] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:38.982839 6393 consensus_queue.cc:1035] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.991736 4644 tablet_service.cc:1467] Processing CreateTablet for tablet f11a1f9872714a3a91b1179e0d44a14a (DEFAULT_TABLE table=default.Table_8ce69a229b7b49b5b712f243d5d4fbb0 [id=fc265cca4a8a42a68d5bdf418d20046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.999533 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ca67743c904927a2ad6c8ca099fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.036003 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a1f9872714a3a91b1179e0d44a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.003330 4634 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57760 (request call id 115) took 1920 ms. Trace:
I20240503 02:12:39.005383 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 41531ddcbf66465094d8f06e1116b3be (DEFAULT_TABLE table=default.Table_cf5d681047e040968c0436492ac29ac6 [id=04018dec3d824068ae653291a488118b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.006950 5115 ts_tablet_manager.cc:1888] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.037328 5115 log.cc:1196] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:39.037443 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41531ddcbf66465094d8f06e1116b3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.037740 5115 ts_tablet_manager.cc:1909] T e94031465ce7407dab0f90ad8809ac21 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.026763 4650 tablet_service.cc:1467] Processing CreateTablet for tablet a070840b5b734edb8508900af51a4de1 (DEFAULT_TABLE table=default.Table_f8cf5f90991c472a9fb961b4e6d6829e [id=ed34a88be4634b46aef3782d2e0300f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.029698 6374 consensus_queue.cc:1035] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.038007 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070840b5b734edb8508900af51a4de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.038014 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 584) took 5422 ms. Trace:
I20240503 02:12:39.038642 5115 rpcz_store.cc:276] 0503 02:12:33.615329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.615441 (+ 112us) service_pool.cc:225] Handling call
0503 02:12:38.324474 (+4709033us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:38.329589 (+ 5115us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.918753 (+589164us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.006840 (+ 88087us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.037996 (+ 31156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4702522}
I20240503 02:12:39.038197 4634 rpcz_store.cc:276] 0503 02:12:37.083184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.083244 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:38.223088 (+1139844us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x6160001f1554. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.585936 (+362848us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x616000142c54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.835456 (+249520us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x616000182254. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:38.928560 (+ 93104us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x616000025f54. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9971bb00 00007f1c9d42db7f 00007f1c97db98e6 00007f1c97db9615 00007f1c94be504e 00007f1c94be3f2e 00007f1c94bea405 00007f1c94bead2f 00007f1c94beab58 0000000000a60bcc
0503 02:12:39.003286 (+ 74726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1938060928}
I20240503 02:12:39.039101 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 583) took 5438 ms. Trace:
I20240503 02:12:39.042068 5115 rpcz_store.cc:276] 0503 02:12:33.601185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:33.601311 (+ 126us) service_pool.cc:225] Handling call
0503 02:12:39.038983 (+5437672us) ts_tablet_manager.cc:1120] Deleting tablet dee34be89bda450296b916163bb10c46
0503 02:12:39.038992 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.039085 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5416237}
I20240503 02:12:39.042680 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 602) took 4917 ms. Trace:
I20240503 02:12:39.043460 5115 rpcz_store.cc:276] 0503 02:12:34.125673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.125749 (+ 76us) service_pool.cc:225] Handling call
0503 02:12:39.042558 (+4916809us) ts_tablet_manager.cc:1120] Deleting tablet c3710b653aa24b7ca389aa95d7cdd2fe
0503 02:12:39.042569 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.042664 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4916580}
I20240503 02:12:39.043841 5115 tablet_replica.cc:330] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.044098 5115 raft_consensus.cc:2229] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.044267 5115 raft_consensus.cc:2258] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.044596 5114 ts_tablet_manager.cc:1888] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.044713 5114 log.cc:1196] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3b41dce88919456aa846dbdcf5e2759f
I20240503 02:12:39.044979 5115 ts_tablet_manager.cc:1875] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.045091 5114 ts_tablet_manager.cc:1909] T 3b41dce88919456aa846dbdcf5e2759f P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:39.045858 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 621) took 4289 ms. Trace:
I20240503 02:12:39.045950 5114 rpcz_store.cc:276] 0503 02:12:34.756315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.756363 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:38.945327 (+4188964us) ts_tablet_manager.cc:1120] Deleting tablet 3b41dce88919456aa846dbdcf5e2759f
0503 02:12:38.945338 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:38.970479 (+ 25141us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.044495 (+ 74016us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.045837 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4134054}
I20240503 02:12:39.047165 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b161628065704c798020edbe098844c2 (table default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]) successfully deleted
W20240503 02:12:39.047583 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b080fc5ffc564420a0b7f2b7460ebbbb because the tablet was not found. No further retry: Not found: Tablet not found: b080fc5ffc564420a0b7f2b7460ebbbb
I20240503 02:12:39.047794 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (table default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]) successfully deleted
I20240503 02:12:39.048009 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3b72c20a4b5a4d2ea266f031733d1aab (table default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]) successfully deleted
I20240503 02:12:39.048204 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 53017352b8f34955b116bbfd32f04854 (table default.Table_bb18c51dac8c46b2a5ddecca6ddd34be [id=47856b6b36f74b5589f9acc60162c1e8]) successfully deleted
I20240503 02:12:39.048404 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 5f56701ed6424c60bf1a985eea1936fe (table default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]) successfully deleted
W20240503 02:12:39.048601 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5cd296bb3b574d6cbb0252ad10ada00c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cd296bb3b574d6cbb0252ad10ada00c already in progress: creating tablet
I20240503 02:12:39.048789 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7178760c21904207b6c8317093871f63 (table default.Table_7d5419d4bbf34b67bcc05cf32663c2df [id=07e16e4056424605a70ccfaa25a24613]) successfully deleted
W20240503 02:12:39.050901 4608 leader_election.cc:341] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:39.053511 5103 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
W20240503 02:12:39.053983 4608 leader_election.cc:341] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:39.054185 4608 leader_election.cc:302] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.054281 5103 ts_tablet_manager.cc:1367] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.009s
I20240503 02:12:39.054567 6393 raft_consensus.cc:2726] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:39.055300 5103 raft_consensus.cc:348] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.055507 5103 raft_consensus.cc:374] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.055580 5103 raft_consensus.cc:729] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.055822 5103 consensus_queue.cc:260] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.056347 5103 ts_tablet_manager.cc:1398] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:39.056650 5103 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.058424 5103 tablet_bootstrap.cc:654] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.034822 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 702) took 1201 ms. Trace:
I20240503 02:12:39.053653 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:39.064664 5159 rpcz_store.cc:276] 0503 02:12:37.833060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.837190 (+ 4130us) service_pool.cc:225] Handling call
0503 02:12:38.958090 (+1120900us) ts_tablet_manager.cc:1120] Deleting tablet d84412b28b4c41c2a3ce18702403dcd8
0503 02:12:38.958099 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.034802 (+ 76703us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120671}
I20240503 02:12:39.065258 5159 tablet_replica.cc:330] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.065482 5159 raft_consensus.cc:2229] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.065721 5159 raft_consensus.cc:2258] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.066571 5159 ts_tablet_manager.cc:1875] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.053728 6262 consensus_queue.cc:1035] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20240503 02:12:39.074363 5115 ts_tablet_manager.cc:1888] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.074676 5115 log.cc:1196] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:39.073604 5105 raft_consensus.cc:348] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.075197 5115 ts_tablet_manager.cc:1909] T 78b21a3010a4470288fbdd684ef64e77 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.075196 5105 raft_consensus.cc:374] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.075637 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 603) took 4949 ms. Trace:
I20240503 02:12:39.075748 5115 rpcz_store.cc:276] 0503 02:12:34.126316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.126382 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:39.043798 (+4917416us) ts_tablet_manager.cc:1120] Deleting tablet 78b21a3010a4470288fbdd684ef64e77
0503 02:12:39.043806 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.060653 (+ 16847us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.074259 (+ 13606us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.075613 (+ 1354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4917207}
I20240503 02:12:39.075860 5105 raft_consensus.cc:729] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.076257 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 604) took 4948 ms. Trace:
I20240503 02:12:39.076344 5115 rpcz_store.cc:276] 0503 02:12:34.127574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.127668 (+ 94us) service_pool.cc:225] Handling call
0503 02:12:39.076101 (+4948433us) ts_tablet_manager.cc:1120] Deleting tablet 211dfd4049684f5e8af927b2a19263b7
0503 02:12:39.076112 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.076240 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4948233}
I20240503 02:12:39.076834 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 605) took 4838 ms. Trace:
I20240503 02:12:39.077019 5115 rpcz_store.cc:276] 0503 02:12:34.238144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.238208 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:39.076631 (+4838423us) ts_tablet_manager.cc:1120] Deleting tablet 50e04292742646049f39b6aa29338d75
0503 02:12:39.076641 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.076815 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4838212}
I20240503 02:12:39.077471 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 606) took 4744 ms. Trace:
I20240503 02:12:39.078068 5115 rpcz_store.cc:276] 0503 02:12:34.333254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.333315 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:39.077341 (+4744026us) ts_tablet_manager.cc:1120] Deleting tablet 6e635f283288406aa16b3941be64e777
0503 02:12:39.077353 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.077454 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4743508}
I20240503 02:12:39.077726 5105 consensus_queue.cc:260] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.081207 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 623) took 4257 ms. Trace:
I20240503 02:12:39.081322 5114 rpcz_store.cc:276] 0503 02:12:34.823376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.823441 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:39.081049 (+4257608us) ts_tablet_manager.cc:1120] Deleting tablet cef84b4e6ba24150905994d45ffd0b52
0503 02:12:39.081061 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.081183 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4257372}
I20240503 02:12:39.081581 5114 tablet_replica.cc:330] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:39.081743 5114 raft_consensus.cc:2229] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.081893 5114 raft_consensus.cc:2258] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:39.082110 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6d08e7d17f1644749a9a2e7175b3191a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d08e7d17f1644749a9a2e7175b3191a already in progress: creating tablet
W20240503 02:12:39.082469 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a8c57ddf3d8428e8fa5ffb061dc054b already in progress: creating tablet
W20240503 02:12:39.082643 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 6b15997a94b74ad3808d965acfb5d155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b15997a94b74ad3808d965acfb5d155 already in progress: creating tablet
W20240503 02:12:39.083032 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d84412b28b4c41c2a3ce18702403dcd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d84412b28b4c41c2a3ce18702403dcd8 already in progress: creating tablet
I20240503 02:12:39.085981 5105 ts_tablet_manager.cc:1398] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:12:39.091501 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 607) took 4545 ms. Trace:
I20240503 02:12:39.097479 5105 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.097056 5115 rpcz_store.cc:276] 0503 02:12:34.546314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.547368 (+ 1054us) service_pool.cc:225] Handling call
0503 02:12:39.089987 (+4542619us) ts_tablet_manager.cc:1120] Deleting tablet f5fd27023b3f47f281dfb5474ea3a4c6
0503 02:12:39.090008 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.091477 (+ 1469us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4513214}
I20240503 02:12:39.110901 6353 raft_consensus.cc:482] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.111056 6353 raft_consensus.cc:504] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.112584 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.111569 5105 tablet_bootstrap.cc:654] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.111696 6353 leader_election.cc:288] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:39.113282 4746 leader_election.cc:341] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.115945 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 2f053537029a44d5a519b612d09d19a1 (table default.Table_26db706b59c346f59cb5df08c11c1d22 [id=d9011705ac294bd5832490cfdb4582b9]) successfully deleted
I20240503 02:12:39.117151 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 3b41dce88919456aa846dbdcf5e2759f (table default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]) successfully deleted
I20240503 02:12:39.112124 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.119057 5115 tablet_replica.cc:330] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.119256 5115 raft_consensus.cc:2229] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.119434 5115 raft_consensus.cc:2258] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:39.119413 4746 leader_election.cc:341] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.119526 4746 leader_election.cc:302] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.119980 6353 raft_consensus.cc:2726] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.120059 5115 ts_tablet_manager.cc:1875] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.114792 5106 tablet_bootstrap.cc:492] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.120344 5106 ts_tablet_manager.cc:1367] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.259s user 0.001s sys 0.002s
I20240503 02:12:39.121367 5106 raft_consensus.cc:348] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.121600 5106 raft_consensus.cc:374] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.121716 5106 raft_consensus.cc:729] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.121963 5106 consensus_queue.cc:260] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.124538 5106 ts_tablet_manager.cc:1398] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:39.125078 5106 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:39.125150 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet cef84b4e6ba24150905994d45ffd0b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef84b4e6ba24150905994d45ffd0b52 already in progress: creating tablet
I20240503 02:12:39.126564 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet e94031465ce7407dab0f90ad8809ac21 (table default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]) successfully deleted
W20240503 02:12:39.127019 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet dee34be89bda450296b916163bb10c46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dee34be89bda450296b916163bb10c46 already in progress: creating tablet
I20240503 02:12:39.127977 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a4a16052d43d45aeb405c5de5dafa041
W20240503 02:12:39.128545 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c3710b653aa24b7ca389aa95d7cdd2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3710b653aa24b7ca389aa95d7cdd2fe already in progress: creating tablet
I20240503 02:12:39.129065 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:39.133596 5114 ts_tablet_manager.cc:1875] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.130383 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 78b21a3010a4470288fbdd684ef64e77 (table default.Table_7566b876f57547599a98c4f55fbfc427 [id=5d52bb117988409d974ea961f9f7bab9]) successfully deleted
W20240503 02:12:39.136958 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d40ed707ad074e0fbb0ee48c2137c661 consensus state has no leader
W20240503 02:12:39.137400 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 211dfd4049684f5e8af927b2a19263b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211dfd4049684f5e8af927b2a19263b7 already in progress: creating tablet
I20240503 02:12:39.130440 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 16b3d1e7ce83425db0bd56457d9f2544
W20240503 02:12:39.138710 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 50e04292742646049f39b6aa29338d75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e04292742646049f39b6aa29338d75 already in progress: creating tablet
I20240503 02:12:39.139084 5106 tablet_bootstrap.cc:654] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.139045 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 16b3d1e7ce83425db0bd56457d9f2544 deleted and replaced by tablet 95c7e4a2954e4161b9acc029873f923b
I20240503 02:12:39.139261 5105 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:39.139768 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6e635f283288406aa16b3941be64e777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e635f283288406aa16b3941be64e777 already in progress: creating tablet
I20240503 02:12:39.140276 5105 ts_tablet_manager.cc:1367] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.002s
W20240503 02:12:39.141455 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f5fd27023b3f47f281dfb5474ea3a4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5fd27023b3f47f281dfb5474ea3a4c6 already in progress: creating tablet
I20240503 02:12:39.142060 5159 ts_tablet_manager.cc:1888] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.142736 5159 log.cc:1196] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b91abaf51abb4c2ca65412c3771c1ef2
I20240503 02:12:39.130762 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a3374e095846463a859bb642c2f42167
I20240503 02:12:39.145066 5159 ts_tablet_manager.cc:1909] T b91abaf51abb4c2ca65412c3771c1ef2 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.145679 5105 raft_consensus.cc:348] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.146481 5105 raft_consensus.cc:374] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.146798 5105 raft_consensus.cc:729] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.147464 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a16052d43d45aeb405c5de5dafa041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.147619 5105 consensus_queue.cc:260] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.148254 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 703) took 1310 ms. Trace:
I20240503 02:12:39.148419 5159 rpcz_store.cc:276] 0503 02:12:37.837986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.840379 (+ 2393us) service_pool.cc:225] Handling call
0503 02:12:39.065207 (+1224828us) ts_tablet_manager.cc:1120] Deleting tablet b91abaf51abb4c2ca65412c3771c1ef2
0503 02:12:39.065219 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.108149 (+ 42930us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.141225 (+ 33076us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.148146 (+ 6921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224639}
I20240503 02:12:39.149792 5159 tablet_replica.cc:330] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.149740 5105 ts_tablet_manager.cc:1398] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:39.150907 5159 raft_consensus.cc:2229] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.151129 5105 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.151497 5159 raft_consensus.cc:2258] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.152813 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a16052d43d45aeb405c5de5dafa041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.153110 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a16052d43d45aeb405c5de5dafa041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.154071 6262 consensus_queue.cc:1035] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.156498 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 with delete_type TABLET_DATA_DELETED (replaced by tablet 95c7e4a2954e4161b9acc029873f923b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.156781 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b91abaf51abb4c2ca65412c3771c1ef2 (table default.Table_df633d54f27d4e71b052cbf0582d8ba2 [id=65c98763e63d4447b90a16f448bccffe]) successfully deleted
I20240503 02:12:39.156886 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet a3374e095846463a859bb642c2f42167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.157756 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2fb716cd099049f99df15bdb22fe90fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.158202 5105 tablet_bootstrap.cc:654] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:39.158455 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb716cd099049f99df15bdb22fe90fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.158347 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6b85445f36e344a8863969b82f94e867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.158831 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b85445f36e344a8863969b82f94e867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.160964 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_86543c68f82543d1bce59d92c6138b7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.161130 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet a3374e095846463a859bb642c2f42167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:39.161461 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86543c68f82543d1bce59d92c6138b7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.162874 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e474067bd616474aa34cb102a02b6b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.163537 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e474067bd616474aa34cb102a02b6b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.165469 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3a2b0f0f386845c18712a0ddb23d9cd0" } new_table_name: "default.Table_d00d4724cb9b445e99d6acb39120559e" modify_external_catalogs: true
I20240503 02:12:39.166752 6296 raft_consensus.cc:482] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.166906 6296 raft_consensus.cc:504] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.167682 6296 leader_election.cc:288] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.168901 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:39.169613 4746 leader_election.cc:341] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.169709 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.170575 4746 leader_election.cc:341] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.170691 4746 leader_election.cc:302] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.171092 6354 raft_consensus.cc:482] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.171231 6354 raft_consensus.cc:504] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.171940 6354 leader_election.cc:288] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.172224 5114 ts_tablet_manager.cc:1888] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.172297 6354 raft_consensus.cc:2726] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.172638 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.176352 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.176553 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.176645 5105 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:39.177285 4746 leader_election.cc:341] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.177264 5105 ts_tablet_manager.cc:1367] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240503 02:12:39.178253 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 with delete_type TABLET_DATA_DELETED (replaced by tablet 95c7e4a2954e4161b9acc029873f923b) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:39.179535 4746 leader_election.cc:341] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.179675 4746 leader_election.cc:302] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.180296 6353 raft_consensus.cc:2726] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.180680 5103 tablet_bootstrap.cc:492] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.180842 5103 ts_tablet_manager.cc:1367] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.004s
I20240503 02:12:39.181671 5103 raft_consensus.cc:348] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.181883 5103 raft_consensus.cc:374] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.181983 5103 raft_consensus.cc:729] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.182272 5103 consensus_queue.cc:260] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.182027 5105 raft_consensus.cc:348] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.183123 5103 ts_tablet_manager.cc:1398] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:39.183457 5103 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.183162 5105 raft_consensus.cc:374] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.183717 5105 raft_consensus.cc:729] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.179613 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.185060 5105 consensus_queue.cc:260] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.188416 5103 tablet_bootstrap.cc:654] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.188606 5115 ts_tablet_manager.cc:1888] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.172334 5114 log.cc:1196] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/d022b5df4a914e439fff918f70c34157
I20240503 02:12:39.188731 5115 log.cc:1196] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/396b48865e954d05b9ba8084a2e7f29b
I20240503 02:12:39.189191 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.190210 5114 ts_tablet_manager.cc:1909] T d022b5df4a914e439fff918f70c34157 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:39.190312 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 with delete_type TABLET_DATA_DELETED (replaced by tablet 95c7e4a2954e4161b9acc029873f923b) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.190999 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet a3374e095846463a859bb642c2f42167 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.191190 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 624) took 4312 ms. Trace:
I20240503 02:12:39.191466 5114 rpcz_store.cc:276] 0503 02:12:34.878629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.879058 (+ 429us) service_pool.cc:225] Handling call
0503 02:12:39.081540 (+4202482us) ts_tablet_manager.cc:1120] Deleting tablet d022b5df4a914e439fff918f70c34157
0503 02:12:39.081548 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.147081 (+ 65533us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.172099 (+ 25018us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.191171 (+ 19072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4202220}
I20240503 02:12:39.191984 5105 ts_tablet_manager.cc:1398] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.014s user 0.003s sys 0.001s
I20240503 02:12:39.192639 5114 tablet_replica.cc:330] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:39.192849 5114 raft_consensus.cc:2229] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.193106 5114 raft_consensus.cc:2258] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.194730 5106 tablet_bootstrap.cc:492] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.194757 5114 ts_tablet_manager.cc:1875] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.194989 5106 ts_tablet_manager.cc:1367] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.002s
I20240503 02:12:39.197338 6353 raft_consensus.cc:482] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.197525 6353 raft_consensus.cc:504] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.198401 6353 leader_election.cc:288] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.205330 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet d022b5df4a914e439fff918f70c34157 (table default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]) successfully deleted
W20240503 02:12:39.207223 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c9090b11c8a4bf0b4b92523b9431b79 consensus state has no leader
I20240503 02:12:39.207599 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fc67bba49712445db6d784b6ed94dd7e
I20240503 02:12:39.207811 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.208341 4746 leader_election.cc:341] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.209751 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:39.210589 4746 leader_election.cc:341] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.210716 4746 leader_election.cc:302] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.211418 6296 raft_consensus.cc:2726] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.196775 5106 raft_consensus.cc:348] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.213718 5106 raft_consensus.cc:374] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.214123 5106 raft_consensus.cc:729] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.214537 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bba49712445db6d784b6ed94dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.215169 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bba49712445db6d784b6ed94dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.215785 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bba49712445db6d784b6ed94dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.216396 5105 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.215873 5106 consensus_queue.cc:260] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.217221 5115 ts_tablet_manager.cc:1909] T 396b48865e954d05b9ba8084a2e7f29b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.219740 5159 ts_tablet_manager.cc:1875] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.220741 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 612) took 4537 ms. Trace:
I20240503 02:12:39.223901 5106 ts_tablet_manager.cc:1398] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240503 02:12:39.226008 5105 tablet_bootstrap.cc:654] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.226357 5106 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.230028 5106 tablet_bootstrap.cc:654] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.238766 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:39.220862 5115 rpcz_store.cc:276] 0503 02:12:34.683511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:34.683565 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:39.119001 (+4435436us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:39.119012 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.185500 (+ 66488us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.188503 (+ 3003us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.220720 (+ 32217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2944,"tablet-delete.queue_time_us":4435208}
I20240503 02:12:39.240504 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 614) took 4224 ms. Trace:
I20240503 02:12:39.240468 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 396b48865e954d05b9ba8084a2e7f29b (table default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]) successfully deleted
I20240503 02:12:39.240628 5115 rpcz_store.cc:276] 0503 02:12:35.015943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.018233 (+ 2290us) service_pool.cc:225] Handling call
0503 02:12:39.240345 (+4222112us) ts_tablet_manager.cc:1120] Deleting tablet b8474c841cb64373a223ad76b43ecf5f
0503 02:12:39.240371 (+ 26us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.240487 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4221775}
I20240503 02:12:39.241092 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 615) took 4222 ms. Trace:
I20240503 02:12:39.241176 5115 rpcz_store.cc:276] 0503 02:12:35.019149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.019412 (+ 263us) service_pool.cc:225] Handling call
0503 02:12:39.240989 (+4221577us) ts_tablet_manager.cc:1120] Deleting tablet a8ce4b33f90c40c48cf0a538a1a7fde4
0503 02:12:39.240997 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.241077 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4221253}
I20240503 02:12:39.241601 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 616) took 4221 ms. Trace:
I20240503 02:12:39.241683 5115 rpcz_store.cc:276] 0503 02:12:35.020033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.021595 (+ 1562us) service_pool.cc:225] Handling call
0503 02:12:39.241513 (+4219918us) ts_tablet_manager.cc:1120] Deleting tablet 84f6279f16dc42d69e794e3c6586163d
0503 02:12:39.241520 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.241587 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4219672}
W20240503 02:12:39.241888 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b8474c841cb64373a223ad76b43ecf5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8474c841cb64373a223ad76b43ecf5f already in progress: creating tablet
I20240503 02:12:39.242067 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 617) took 4219 ms. Trace:
I20240503 02:12:39.242142 5115 rpcz_store.cc:276] 0503 02:12:35.022923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.022966 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:39.241981 (+4219015us) ts_tablet_manager.cc:1120] Deleting tablet 04d555c8d8844ee49c960a5279485480
0503 02:12:39.241988 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.242054 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4218766}
I20240503 02:12:39.242499 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 618) took 4218 ms. Trace:
I20240503 02:12:39.242583 5115 rpcz_store.cc:276] 0503 02:12:35.023603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.023650 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:39.242412 (+4218762us) ts_tablet_manager.cc:1120] Deleting tablet 8135907b78284d7ca74396cc77dd0612
0503 02:12:39.242420 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.242485 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4218574}
W20240503 02:12:39.242640 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ce4b33f90c40c48cf0a538a1a7fde4 already in progress: creating tablet
I20240503 02:12:39.242957 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 619) took 4218 ms. Trace:
I20240503 02:12:39.243033 5115 rpcz_store.cc:276] 0503 02:12:35.024032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.028898 (+ 4866us) service_pool.cc:225] Handling call
0503 02:12:39.242872 (+4213974us) ts_tablet_manager.cc:1120] Deleting tablet 7447242714c844e685faca5aed71a7f7
0503 02:12:39.242879 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.242941 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4213632}
I20240503 02:12:39.243427 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 620) took 4213 ms. Trace:
I20240503 02:12:39.243520 5115 rpcz_store.cc:276] 0503 02:12:35.029560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.029612 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:39.243323 (+4213711us) ts_tablet_manager.cc:1120] Deleting tablet cef84b4e6ba24150905994d45ffd0b52
0503 02:12:39.243331 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.243411 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4213525}
W20240503 02:12:39.243844 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 04d555c8d8844ee49c960a5279485480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04d555c8d8844ee49c960a5279485480 already in progress: creating tablet
W20240503 02:12:39.244330 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8135907b78284d7ca74396cc77dd0612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8135907b78284d7ca74396cc77dd0612 already in progress: creating tablet
I20240503 02:12:39.244820 5103 tablet_bootstrap.cc:492] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.244983 5103 ts_tablet_manager.cc:1367] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
W20240503 02:12:39.245236 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7447242714c844e685faca5aed71a7f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7447242714c844e685faca5aed71a7f7 already in progress: creating tablet
I20240503 02:12:39.245883 5103 raft_consensus.cc:348] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.246348 5103 raft_consensus.cc:374] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.246434 5103 raft_consensus.cc:729] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
W20240503 02:12:39.246383 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet cef84b4e6ba24150905994d45ffd0b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef84b4e6ba24150905994d45ffd0b52 already in progress: creating tablet
I20240503 02:12:39.246631 5106 tablet_bootstrap.cc:492] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.246663 5103 consensus_queue.cc:260] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.246795 5106 ts_tablet_manager.cc:1367] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
I20240503 02:12:39.247648 5106 raft_consensus.cc:348] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.247895 5106 raft_consensus.cc:374] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.247995 5106 raft_consensus.cc:729] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.248322 5106 consensus_queue.cc:260] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.249006 6262 raft_consensus.cc:482] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.249114 6262 raft_consensus.cc:504] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.249682 6262 leader_election.cc:288] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.250385 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.250571 4518 raft_consensus.cc:2454] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:39.251214 4608 leader_election.cc:302] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.251137 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.251399 4811 raft_consensus.cc:2454] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:39.251603 6262 raft_consensus.cc:2781] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.251684 6262 raft_consensus.cc:482] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.251745 6262 raft_consensus.cc:3037] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.252218 5103 ts_tablet_manager.cc:1398] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:39.253024 5103 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.254451 5115 tablet_replica.cc:330] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.254376 6262 raft_consensus.cc:504] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.254946 6262 leader_election.cc:288] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.255846 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:39.256031 4518 raft_consensus.cc:3037] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.258787 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74f4816cad04a9eb5934aaba86c2fbc" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.259018 4811 raft_consensus.cc:3037] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.259078 4518 raft_consensus.cc:2454] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:39.259159 5115 raft_consensus.cc:2229] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.259507 4608 leader_election.cc:302] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.259908 5106 ts_tablet_manager.cc:1398] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:12:39.260036 6262 raft_consensus.cc:2781] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.260224 5106 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.260249 6262 raft_consensus.cc:686] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:39.261618 6262 consensus_queue.cc:237] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.260538 5115 raft_consensus.cc:2258] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.263953 5115 ts_tablet_manager.cc:1875] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.267692 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:39.268273 5159 ts_tablet_manager.cc:1888] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.268383 5159 log.cc:1196] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/41ef3c6e5e7d40249b112144975ba220
I20240503 02:12:39.269389 5105 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.269582 5105 ts_tablet_manager.cc:1367] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20240503 02:12:39.270599 5105 raft_consensus.cc:348] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.270828 5105 raft_consensus.cc:374] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.270910 5105 raft_consensus.cc:729] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.271129 5105 consensus_queue.cc:260] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.271672 5105 ts_tablet_manager.cc:1398] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:39.273561 5105 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.273546 5868 raft_consensus.cc:482] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.273672 5868 raft_consensus.cc:504] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.274281 5868 leader_election.cc:288] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:39.274873 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.274938 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.275065 4672 raft_consensus.cc:2454] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.275117 4808 raft_consensus.cc:2454] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.275393 5105 tablet_bootstrap.cc:654] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.275512 4462 leader_election.cc:302] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.275686 5103 tablet_bootstrap.cc:654] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.275972 5868 raft_consensus.cc:2781] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.276050 5868 raft_consensus.cc:482] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.276115 5868 raft_consensus.cc:3037] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.276541 6448 raft_consensus.cc:482] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.275873 5159 ts_tablet_manager.cc:1909] T 41ef3c6e5e7d40249b112144975ba220 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.276652 6448 raft_consensus.cc:504] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.277009 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 704) took 1436 ms. Trace:
I20240503 02:12:39.277374 5106 tablet_bootstrap.cc:654] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.277133 5159 rpcz_store.cc:276] 0503 02:12:37.840700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.840872 (+ 172us) service_pool.cc:225] Handling call
0503 02:12:39.149647 (+1308775us) ts_tablet_manager.cc:1120] Deleting tablet 41ef3c6e5e7d40249b112144975ba220
0503 02:12:39.149659 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.227479 (+ 77820us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.268155 (+ 40676us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.276991 (+ 8836us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62301,"tablet-delete.queue_time_us":1308325}
I20240503 02:12:39.277431 6448 leader_election.cc:288] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:39.277592 5159 tablet_replica.cc:330] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.277818 5159 raft_consensus.cc:2229] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.277997 5159 raft_consensus.cc:2258] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.278780 5159 ts_tablet_manager.cc:1875] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.278889 5868 raft_consensus.cc:504] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.279788 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.279970 4672 raft_consensus.cc:2454] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.280444 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.280521 4462 leader_election.cc:302] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.280618 4808 raft_consensus.cc:2454] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.280938 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 41ef3c6e5e7d40249b112144975ba220 (table default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]) successfully deleted
I20240503 02:12:39.281302 6448 raft_consensus.cc:2781] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.281374 6448 raft_consensus.cc:482] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.281436 6448 raft_consensus.cc:3037] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.283962 6448 raft_consensus.cc:504] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.284572 6448 leader_election.cc:288] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:39.285636 5115 ts_tablet_manager.cc:1888] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:39.285735 5115 log.cc:1196] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/17afd88bf6c040ad9e8d571eccd348dc
I20240503 02:12:39.286428 5115 ts_tablet_manager.cc:1909] T 17afd88bf6c040ad9e8d571eccd348dc P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.286763 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 621) took 4176 ms. Trace:
I20240503 02:12:39.286844 5115 rpcz_store.cc:276] 0503 02:12:35.110230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.110298 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:39.254393 (+4144095us) ts_tablet_manager.cc:1120] Deleting tablet 17afd88bf6c040ad9e8d571eccd348dc
0503 02:12:39.254404 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.272346 (+ 17942us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.285528 (+ 13182us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.286743 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4143782}
I20240503 02:12:39.287905 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:39.288079 4672 raft_consensus.cc:3037] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.288137 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 623) took 4038 ms. Trace:
I20240503 02:12:39.288203 5115 rpcz_store.cc:276] 0503 02:12:35.249255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.249320 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:39.287943 (+4038623us) ts_tablet_manager.cc:1120] Deleting tablet 6ecdb634fb1c4337b2eae0c844fbb1dc
0503 02:12:39.287950 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.288113 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4038267}
I20240503 02:12:39.288727 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 624) took 3998 ms. Trace:
I20240503 02:12:39.288790 5115 rpcz_store.cc:276] 0503 02:12:35.290736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.290787 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:39.288638 (+3997851us) ts_tablet_manager.cc:1120] Deleting tablet ee1fac803dd84eee8722f0a14559faf6
0503 02:12:39.288646 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.288715 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3997575}
I20240503 02:12:39.288834 4811 raft_consensus.cc:2454] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:39.289579 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 625) took 3990 ms. Trace:
I20240503 02:12:39.289647 5115 rpcz_store.cc:276] 0503 02:12:35.298697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.299165 (+ 468us) service_pool.cc:225] Handling call
0503 02:12:39.289490 (+3990325us) ts_tablet_manager.cc:1120] Deleting tablet 9290a61c21fa456eb3c4f9b0dbebc864
0503 02:12:39.289499 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.289565 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3990004}
I20240503 02:12:39.290377 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 636) took 3749 ms. Trace:
I20240503 02:12:39.290443 5115 rpcz_store.cc:276] 0503 02:12:35.541417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.541493 (+ 76us) service_pool.cc:225] Handling call
0503 02:12:39.290286 (+3748793us) ts_tablet_manager.cc:1120] Deleting tablet 8c6c80efe35343989f46a2c181f38129
0503 02:12:39.290294 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.290363 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3748479}
I20240503 02:12:39.290570 5159 ts_tablet_manager.cc:1888] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.290663 5159 log.cc:1196] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/f2f9b962c9264f87b798df33544cb15d
I20240503 02:12:39.291016 5159 ts_tablet_manager.cc:1909] T f2f9b962c9264f87b798df33544cb15d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.290978 4672 raft_consensus.cc:2454] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.291227 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 637) took 3725 ms. Trace:
I20240503 02:12:39.291301 5115 rpcz_store.cc:276] 0503 02:12:35.565864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.565932 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:39.291122 (+3725190us) ts_tablet_manager.cc:1120] Deleting tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2
0503 02:12:39.291143 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.291215 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3724984}
I20240503 02:12:39.291580 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 638) took 3712 ms. Trace:
I20240503 02:12:39.291750 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 705) took 1450 ms. Trace:
I20240503 02:12:39.292193 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 17afd88bf6c040ad9e8d571eccd348dc (table default.Table_8b93d7e7be90441ea960cdf3b81e4fc7 [id=5febc608abd84e7ba26288f2e155d449]) successfully deleted
W20240503 02:12:39.292557 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecdb634fb1c4337b2eae0c844fbb1dc already in progress: creating tablet
I20240503 02:12:39.291647 5115 rpcz_store.cc:276] 0503 02:12:35.578948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.579017 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:39.291492 (+3712475us) ts_tablet_manager.cc:1120] Deleting tablet 3e94a9c61cb74cc08ca5867ae85ecdf0
0503 02:12:39.291499 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.291566 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3712213}
W20240503 02:12:39.292840 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ee1fac803dd84eee8722f0a14559faf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1fac803dd84eee8722f0a14559faf6 already in progress: creating tablet
I20240503 02:12:39.293100 4462 leader_election.cc:302] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:39.293596 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9290a61c21fa456eb3c4f9b0dbebc864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9290a61c21fa456eb3c4f9b0dbebc864 already in progress: creating tablet
I20240503 02:12:39.293785 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 639) took 3705 ms. Trace:
W20240503 02:12:39.293845 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8c6c80efe35343989f46a2c181f38129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6c80efe35343989f46a2c181f38129 already in progress: creating tablet
I20240503 02:12:39.293864 5115 rpcz_store.cc:276] 0503 02:12:35.588837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.588934 (+ 97us) service_pool.cc:225] Handling call
0503 02:12:39.293336 (+3704402us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:39.293346 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.293769 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704137}
W20240503 02:12:39.294027 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 already in progress: creating tablet
W20240503 02:12:39.294273 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 already in progress: creating tablet
I20240503 02:12:39.294508 5115 tablet_replica.cc:330] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:39.294508 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 65774b69a8bd4664a6b55b84eba37f7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65774b69a8bd4664a6b55b84eba37f7f already in progress: creating tablet
I20240503 02:12:39.295083 5115 raft_consensus.cc:2229] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.295408 5115 raft_consensus.cc:2258] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.296505 5115 ts_tablet_manager.cc:1875] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.296221 6448 raft_consensus.cc:2781] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.297286 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:39.298175 4672 raft_consensus.cc:3037] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.298579 6448 raft_consensus.cc:686] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:39.299690 5868 leader_election.cc:288] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:39.300349 5114 ts_tablet_manager.cc:1888] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:39.301517 5106 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.301699 5106 ts_tablet_manager.cc:1367] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.002s
I20240503 02:12:39.302248 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8d03de02b1164ff5b180939da7d95ad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.302806 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d03de02b1164ff5b180939da7d95ad3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.302830 5106 raft_consensus.cc:348] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.303162 5106 raft_consensus.cc:374] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.303256 5106 raft_consensus.cc:729] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.303534 5106 consensus_queue.cc:260] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.303664 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b94fa9b463444b4699a8781bcdca5f35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.304098 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b94fa9b463444b4699a8781bcdca5f35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.311923 4270 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c9090b11c8a4bf0b4b92523b9431b79 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:39.312398 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90128cc31d5a49fb8318274b7ced27f2 consensus state has no leader
I20240503 02:12:39.293211 5105 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.312690 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet c3710b653aa24b7ca389aa95d7cdd2fe from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:39.294065 6353 raft_consensus.cc:482] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.312965 6353 raft_consensus.cc:504] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.312944 5159 rpcz_store.cc:276] 0503 02:12:37.841157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.842315 (+ 1158us) service_pool.cc:225] Handling call
0503 02:12:39.277546 (+1435231us) ts_tablet_manager.cc:1120] Deleting tablet f2f9b962c9264f87b798df33544cb15d
0503 02:12:39.277556 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.287486 (+ 9930us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.290481 (+ 2995us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.291733 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435075}
I20240503 02:12:39.313158 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bb63ba232a074c0692fa657db43e25dc" } modify_external_catalogs: true
I20240503 02:12:39.313361 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 706) took 1470 ms. Trace:
I20240503 02:12:39.313330 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bb63ba232a074c0692fa657db43e25dc" } modify_external_catalogs: true
I20240503 02:12:39.313387 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_269475386cb84ba993cddceff2c15394" } new_table_name: "default.Table_91f922f5b0fe4ed99e6be43289e11b8f" modify_external_catalogs: true
I20240503 02:12:39.313442 5159 rpcz_store.cc:276] 0503 02:12:37.842599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.842764 (+ 165us) service_pool.cc:225] Handling call
0503 02:12:39.313260 (+1470496us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:39.313271 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.313345 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470343}
I20240503 02:12:39.313620 6353 leader_election.cc:288] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.299635 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7afb429d64497abcd9f1d0910651c3" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.313696 5159 tablet_replica.cc:330] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.312800 5105 ts_tablet_manager.cc:1367] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240503 02:12:39.313788 4810 raft_consensus.cc:3037] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.314071 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3cc52a664da0462f93bc3453acd53989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.314572 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cc52a664da0462f93bc3453acd53989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.314534 5105 raft_consensus.cc:348] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.314692 5105 raft_consensus.cc:374] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.314734 5105 raft_consensus.cc:729] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.314905 5105 consensus_queue.cc:260] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.315342 5105 ts_tablet_manager.cc:1398] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:39.315613 5105 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.315666 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_73dc468aacee46c397690de4bffb0c0b" } new_table_name: "default.Table_6331571513fc404ba44a641a7af82c0a" modify_external_catalogs: true
I20240503 02:12:39.316457 4810 raft_consensus.cc:2454] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.305469 6296 raft_consensus.cc:482] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.316892 6296 raft_consensus.cc:504] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.317284 5105 tablet_bootstrap.cc:654] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.317523 6296 leader_election.cc:288] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.319507 5159 raft_consensus.cc:2229] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.319993 5159 raft_consensus.cc:2258] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.320729 5159 ts_tablet_manager.cc:1875] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.321946 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet f2f9b962c9264f87b798df33544cb15d (table default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]) successfully deleted
I20240503 02:12:39.295773 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e366ce9d79b3458fb26e96b6e9278fbe" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.323382 4808 raft_consensus.cc:3037] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:39.323148 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 because the tablet was not found. No further retry: Not found: Tablet not found: 3e6a3ca705e8482a8cd6a6fb17d93470
I20240503 02:12:39.306279 5106 ts_tablet_manager.cc:1398] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:39.325995 4808 raft_consensus.cc:2454] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.329723 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 868a6ad6cc1a40f091b4516676c24c97 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:39.337505 5106 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.339025 5105 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.339241 5105 ts_tablet_manager.cc:1367] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240503 02:12:39.340085 5105 raft_consensus.cc:348] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.340317 5105 raft_consensus.cc:374] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.340389 5105 raft_consensus.cc:729] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.340593 5105 consensus_queue.cc:260] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.341127 5105 ts_tablet_manager.cc:1398] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:39.341439 5105 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.342561 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3b9a888b894897b7cc3cf08402febd (DEFAULT_TABLE table=default.Table_8e334557d1ca4d7faa91def038f0afc4 [id=4ddec7814f444b16bb526e1bef1c36bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.342952 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd60af7cb64702aca0d417d2c27efd (DEFAULT_TABLE table=default.Table_2184d913525e433d87c1fc3346faf9d7 [id=59885a418c3a423b9f89ab89cd34bccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.343199 5105 tablet_bootstrap.cc:654] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.343729 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3b9a888b894897b7cc3cf08402febd (DEFAULT_TABLE table=default.Table_8e334557d1ca4d7faa91def038f0afc4 [id=4ddec7814f444b16bb526e1bef1c36bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.318284 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9090b11c8a4bf0b4b92523b9431b79" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.344167 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3b9a888b894897b7cc3cf08402febd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.344540 5106 tablet_bootstrap.cc:654] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.345459 5114 log.cc:1196] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/17afd88bf6c040ad9e8d571eccd348dc
I20240503 02:12:39.345911 5114 ts_tablet_manager.cc:1909] T 17afd88bf6c040ad9e8d571eccd348dc P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:39.349027 4644 tablet_service.cc:1467] Processing CreateTablet for tablet eb50b5963447493e8d8db2130feaa9bd (DEFAULT_TABLE table=default.Table_73dc468aacee46c397690de4bffb0c0b [id=6e9950e1924e4075a5e6a6e80e0f6de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.349426 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb50b5963447493e8d8db2130feaa9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.343729 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd60af7cb64702aca0d417d2c27efd (DEFAULT_TABLE table=default.Table_2184d913525e433d87c1fc3346faf9d7 [id=59885a418c3a423b9f89ab89cd34bccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.351677 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd60af7cb64702aca0d417d2c27efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.355175 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 (DEFAULT_TABLE table=default.Table_3755b2f2996b4bc785bf23c4c7af2cf2 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.355544 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972cd318ebed41c0ab1b8705f52bdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.357239 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 (DEFAULT_TABLE table=default.Table_3755b2f2996b4bc785bf23c4c7af2cf2 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.357650 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972cd318ebed41c0ab1b8705f52bdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.357738 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 625) took 4259 ms. Trace:
I20240503 02:12:39.357828 5114 rpcz_store.cc:276] 0503 02:12:35.098710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.098785 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:39.192589 (+4093804us) ts_tablet_manager.cc:1120] Deleting tablet 17afd88bf6c040ad9e8d571eccd348dc
0503 02:12:39.192599 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.203086 (+ 10487us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.300245 (+ 97159us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.357716 (+ 57471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4093372}
I20240503 02:12:39.358273 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 17afd88bf6c040ad9e8d571eccd348dc (table default.Table_8b93d7e7be90441ea960cdf3b81e4fc7 [id=5febc608abd84e7ba26288f2e155d449]) successfully deleted
I20240503 02:12:39.360083 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 3f713d4ea7d145e89ca29a151208e04b (DEFAULT_TABLE table=default.Table_ab5717cf4ced4b728fad84ef05751e14 [id=21d901f59b584d10b3d3334f9fcc641b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.319622 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.360468 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f713d4ea7d145e89ca29a151208e04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.336606 4269 catalog_manager.cc:5484] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:39.365245 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]). Expected version 1 got 0
I20240503 02:12:39.345258 4788 tablet_service.cc:1467] Processing CreateTablet for tablet eb50b5963447493e8d8db2130feaa9bd (DEFAULT_TABLE table=default.Table_73dc468aacee46c397690de4bffb0c0b [id=6e9950e1924e4075a5e6a6e80e0f6de4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:39.353367 4746 leader_election.cc:341] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.354270 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 (DEFAULT_TABLE table=default.Table_3755b2f2996b4bc785bf23c4c7af2cf2 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.365869 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 001fe4bb80c24d8e97d971a7130b2296 (table default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2]). Expected version 2 got 0
I20240503 02:12:39.365913 5106 tablet_bootstrap.cc:492] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.366271 5106 ts_tablet_manager.cc:1367] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.003s
I20240503 02:12:39.366317 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
W20240503 02:12:39.366371 4746 leader_election.cc:341] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.366704 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_da717b1e145649f3b0762f6ba74ae5f5 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 1 got 0
I20240503 02:12:39.367110 4269 catalog_manager.cc:5484] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:39.367595 4269 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:39.367945 5106 raft_consensus.cc:348] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.368389 5106 raft_consensus.cc:374] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.300835 6448 consensus_queue.cc:237] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.318857 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9090b11c8a4bf0b4b92523b9431b79" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.368672 5106 raft_consensus.cc:729] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.368974 5159 ts_tablet_manager.cc:1888] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.319238 4462 leader_election.cc:302] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:39.369407 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 50199.9us
I20240503 02:12:39.369158 5106 consensus_queue.cc:260] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.320102 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.337527 4493 tablet_service.cc:1467] Processing CreateTablet for tablet eb50b5963447493e8d8db2130feaa9bd (DEFAULT_TABLE table=default.Table_73dc468aacee46c397690de4bffb0c0b [id=6e9950e1924e4075a5e6a6e80e0f6de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.371305 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb50b5963447493e8d8db2130feaa9bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:39.369131 4746 leader_election.cc:341] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.371397 4746 leader_election.cc:302] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.371631 5106 ts_tablet_manager.cc:1398] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
W20240503 02:12:39.372156 4746 leader_election.cc:341] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.372284 4746 leader_election.cc:302] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.339530 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3b9a888b894897b7cc3cf08402febd (DEFAULT_TABLE table=default.Table_8e334557d1ca4d7faa91def038f0afc4 [id=4ddec7814f444b16bb526e1bef1c36bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.373242 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3b9a888b894897b7cc3cf08402febd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.373615 5106 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.375458 6349 raft_consensus.cc:2726] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.375658 6349 raft_consensus.cc:2726] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.339926 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd60af7cb64702aca0d417d2c27efd (DEFAULT_TABLE table=default.Table_2184d913525e433d87c1fc3346faf9d7 [id=59885a418c3a423b9f89ab89cd34bccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.377214 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd60af7cb64702aca0d417d2c27efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.377888 5159 log.cc:1196] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a74f4816cad04a9eb5934aaba86c2fbc
I20240503 02:12:39.378295 5159 ts_tablet_manager.cc:1909] T a74f4816cad04a9eb5934aaba86c2fbc P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:39.378716 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb50b5963447493e8d8db2130feaa9bd consensus state has no leader
I20240503 02:12:39.378940 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 1716c4807e004760b34c0492cb7f522d from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:39.380266 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_56a4b41e0c25434c814a607b0e6450b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.380790 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56a4b41e0c25434c814a607b0e6450b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.381721 5105 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:39.381839 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50b60601e87d4a1f9ab9b92099df5ca5 consensus state has no leader
I20240503 02:12:39.381850 5105 ts_tablet_manager.cc:1367] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240503 02:12:39.382158 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f9e55262006461d8fd7dd706e640469" } modify_external_catalogs: true
I20240503 02:12:39.382321 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4f9e55262006461d8fd7dd706e640469" } modify_external_catalogs: true
I20240503 02:12:39.382619 5105 raft_consensus.cc:348] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.382808 5105 raft_consensus.cc:374] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.382867 5105 raft_consensus.cc:729] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.383082 5105 consensus_queue.cc:260] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:39.384011 4608 consensus_peers.cc:485] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ced4dfb2cf541af98d50d22d1a0f0ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:39.384218 5105 ts_tablet_manager.cc:1398] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:39.384908 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 868a6ad6cc1a40f091b4516676c24c97
I20240503 02:12:39.385051 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4225c1bc133d4e18a4a47fd58820ca10
W20240503 02:12:39.385406 4462 consensus_peers.cc:485] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b72c20a4b5a4d2ea266f031733d1aab. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:39.385447 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 868a6ad6cc1a40f091b4516676c24c97 deleted and replaced by tablet d93e0ae90b5444228800872b90d94581
I20240503 02:12:39.385747 6317 consensus_queue.cc:1035] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.385777 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b1103caea8f04d32ba60345676c5fa63" } modify_external_catalogs: true
I20240503 02:12:39.385890 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b2cd99be75e54340bb07407e8831e25c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.385977 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b1103caea8f04d32ba60345676c5fa63" } modify_external_catalogs: true
W20240503 02:12:39.386430 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2cd99be75e54340bb07407e8831e25c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.387719 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 6c9090b11c8a4bf0b4b92523b9431b79
I20240503 02:12:39.387995 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 with delete_type TABLET_DATA_DELETED (replaced by tablet d93e0ae90b5444228800872b90d94581) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.388164 4270 catalog_manager.cc:6554] ReplaceTablet: tablet 6c9090b11c8a4bf0b4b92523b9431b79 deleted and replaced by tablet d7ba9a44d78b4a649a6d358e5c187974
I20240503 02:12:39.388473 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_da717b1e145649f3b0762f6ba74ae5f5" } new_table_name: "default.Table_2df5b1aa30a0494f86117be55903a129" modify_external_catalogs: true
I20240503 02:12:39.388952 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.389611 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 with delete_type TABLET_DATA_DELETED (replaced by tablet d93e0ae90b5444228800872b90d94581) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.389794 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.390395 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4225c1bc133d4e18a4a47fd58820ca10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.391173 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 with delete_type TABLET_DATA_DELETED (replaced by tablet d7ba9a44d78b4a649a6d358e5c187974) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.391844 5868 raft_consensus.cc:2781] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.391955 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 with delete_type TABLET_DATA_DELETED (replaced by tablet d7ba9a44d78b4a649a6d358e5c187974) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.392603 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3fabf651e344476ca18d32ef633dd619"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.393209 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fabf651e344476ca18d32ef633dd619 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.393946 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 868a6ad6cc1a40f091b4516676c24c97 with delete_type TABLET_DATA_DELETED (replaced by tablet d93e0ae90b5444228800872b90d94581) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.397037 5114 tablet_replica.cc:330] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:39.397235 5114 raft_consensus.cc:2229] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.397346 5103 tablet_bootstrap.cc:492] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.397384 5114 raft_consensus.cc:2258] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.397470 5103 ts_tablet_manager.cc:1367] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.145s user 0.003s sys 0.000s
I20240503 02:12:39.398864 5103 raft_consensus.cc:348] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.399298 5103 raft_consensus.cc:374] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.399392 5103 raft_consensus.cc:729] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.399659 5103 consensus_queue.cc:260] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.400743 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3b9a888b894897b7cc3cf08402febd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.397871 5114 ts_tablet_manager.cc:1875] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.404028 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 3f713d4ea7d145e89ca29a151208e04b (DEFAULT_TABLE table=default.Table_ab5717cf4ced4b728fad84ef05751e14 [id=21d901f59b584d10b3d3334f9fcc641b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.404367 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f713d4ea7d145e89ca29a151208e04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.404393 6296 raft_consensus.cc:482] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.404511 6296 raft_consensus.cc:504] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.405102 6296 leader_election.cc:288] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.405730 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.405819 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd60af7cb64702aca0d417d2c27efd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:39.406450 4746 leader_election.cc:341] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.406883 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.407300 4746 leader_election.cc:341] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.407390 4746 leader_election.cc:302] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.407858 6296 raft_consensus.cc:2726] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.409099 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb50b5963447493e8d8db2130feaa9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.410224 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9090b11c8a4bf0b4b92523b9431b79 with delete_type TABLET_DATA_DELETED (replaced by tablet d7ba9a44d78b4a649a6d358e5c187974) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.411789 4507 tablet_service.cc:1467] Processing CreateTablet for tablet d99a7e930a3947b4aae62424f11d7227 (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.412127 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99a7e930a3947b4aae62424f11d7227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.413113 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972cd318ebed41c0ab1b8705f52bdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.413553 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 707) took 1570 ms. Trace:
I20240503 02:12:39.413626 5159 rpcz_store.cc:276] 0503 02:12:37.843054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.843209 (+ 155us) service_pool.cc:225] Handling call
0503 02:12:39.313652 (+1470443us) ts_tablet_manager.cc:1120] Deleting tablet a74f4816cad04a9eb5934aaba86c2fbc
0503 02:12:39.313660 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.334767 (+ 21107us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.368838 (+ 34071us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.413534 (+ 44696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470324}
I20240503 02:12:39.413940 5159 tablet_replica.cc:330] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.414100 5159 raft_consensus.cc:2229] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:39.414356 5159 raft_consensus.cc:2258] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.414880 5159 ts_tablet_manager.cc:1875] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.417166 5105 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.418320 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 118ea8bd72bc4d3db475ae4404e35b1b (DEFAULT_TABLE table=default.Table_cb6f41cdb8344d96ba919d9493b64365 [id=d65e3d06276e4e049c631ee3727a2fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.418676 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118ea8bd72bc4d3db475ae4404e35b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.418787 5105 tablet_bootstrap.cc:654] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.421118 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 3f713d4ea7d145e89ca29a151208e04b (DEFAULT_TABLE table=default.Table_ab5717cf4ced4b728fad84ef05751e14 [id=21d901f59b584d10b3d3334f9fcc641b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.421602 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f713d4ea7d145e89ca29a151208e04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.422776 5106 tablet_bootstrap.cc:654] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.423472 5159 ts_tablet_manager.cc:1888] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.425182 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 118ea8bd72bc4d3db475ae4404e35b1b (DEFAULT_TABLE table=default.Table_cb6f41cdb8344d96ba919d9493b64365 [id=d65e3d06276e4e049c631ee3727a2fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.425554 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118ea8bd72bc4d3db475ae4404e35b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.425750 5159 log.cc:1196] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:39.426147 5159 ts_tablet_manager.cc:1909] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.428056 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 708) took 1584 ms. Trace:
I20240503 02:12:39.428655 5159 rpcz_store.cc:276] 0503 02:12:37.843448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.853650 (+ 10202us) service_pool.cc:225] Handling call
0503 02:12:39.413903 (+1560253us) ts_tablet_manager.cc:1120] Deleting tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
0503 02:12:39.413911 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.420494 (+ 6583us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.423390 (+ 2896us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.428037 (+ 4647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559996}
I20240503 02:12:39.429580 5159 tablet_replica.cc:330] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.430856 4644 tablet_service.cc:1467] Processing CreateTablet for tablet d99a7e930a3947b4aae62424f11d7227 (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.431219 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99a7e930a3947b4aae62424f11d7227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.432030 5159 raft_consensus.cc:2229] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.432202 5159 raft_consensus.cc:2258] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.433257 5159 ts_tablet_manager.cc:1875] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.433691 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 118ea8bd72bc4d3db475ae4404e35b1b (DEFAULT_TABLE table=default.Table_cb6f41cdb8344d96ba919d9493b64365 [id=d65e3d06276e4e049c631ee3727a2fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.436770 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118ea8bd72bc4d3db475ae4404e35b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.437368 4781 tablet_service.cc:1467] Processing CreateTablet for tablet d99a7e930a3947b4aae62424f11d7227 (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.437788 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99a7e930a3947b4aae62424f11d7227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.441671 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_9cafcae38e354ce9af1528fa96cdc481 [id=819cdd542d91449db9dd3f7a0c668d95] alter complete (version 3)
W20240503 02:12:39.443120 4271 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:38519 (request call id 88) took 1140 ms (client timeout 1000 ms). Trace:
W20240503 02:12:39.443231 4271 rpcz_store.cc:269] 0503 02:12:38.302875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.355938 (+ 53063us) service_pool.cc:225] Handling call
0503 02:12:39.443039 (+1087101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:38.719246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:38.719407 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:38.719411 (+ 4us) write_op.cc:435] Acquired schema lock
0503 02:12:38.719413 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:38.719553 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:38.719605 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:38.719610 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:38.719735 (+ 125us) tablet.cc:699] Row locks acquired
0503 02:12:38.719739 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:38.719769 (+ 30us) write_op.cc:270] Start()
0503 02:12:38.719820 (+ 51us) write_op.cc:276] Timestamp: P: 1714702358719732 usec, L: 0
0503 02:12:38.719824 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:38.720062 (+ 238us) log.cc:842] Serialized 356 byte log entry
0503 02:12:39.155200 (+435138us) op_driver.cc:444] REPLICATION: finished
0503 02:12:39.206168 (+ 50968us) write_op.cc:301] APPLY: starting
0503 02:12:39.206187 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:39.206231 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:39.206266 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:39.206324 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:39.206360 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:39.206368 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:39.206454 (+ 86us) log.cc:842] Serialized 23 byte log entry
0503 02:12:39.206522 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.206542 (+ 20us) write_op.cc:454] Released schema lock
0503 02:12:39.206550 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:39.206609 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.206613 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":212,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":50909,"apply.run_cpu_time_us":497,"apply.run_wall_time_us":497,"mutex_wait_us":20834,"num_ops":1,"prepare.queue_time_us":24374,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":1222,"replication_time_us":435344}]]}
I20240503 02:12:39.443992 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c569a33635544c3189f27a82ffb3d32f" } modify_external_catalogs: true
I20240503 02:12:39.444177 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c569a33635544c3189f27a82ffb3d32f" } modify_external_catalogs: true
I20240503 02:12:39.448669 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1716c4807e004760b34c0492cb7f522d
I20240503 02:12:39.449159 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.449286 4272 catalog_manager.cc:6554] ReplaceTablet: tablet 1716c4807e004760b34c0492cb7f522d deleted and replaced by tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2
I20240503 02:12:39.449618 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bb24fceb8c3e46a89b5f2be967010467" } new_table_name: "default.Table_8b8fa0fcd56345b59028f0a30d31a5dd" modify_external_catalogs: true
I20240503 02:12:39.450251 4491 tablet_service.cc:1467] Processing CreateTablet for tablet a83d3bf41e8044e7a8f539429e43cdf3 (DEFAULT_TABLE table=default.Table_76c5d3de08164c7c893ba2fe01b1e38b [id=30b54a8aa5aa4241a076c3526d985e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.450304 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a74f4816cad04a9eb5934aaba86c2fbc (table default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]) successfully deleted
I20240503 02:12:39.450624 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83d3bf41e8044e7a8f539429e43cdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.451037 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.451267 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet d022b5df4a914e439fff918f70c34157 with delete_type TABLET_DATA_DELETED (replaced by tablet 20ce11c70a924b8d92d8f36fc7401dfd) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.451711 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.451709 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5920bfd139224d908c23f21d6a65308a" } modify_external_catalogs: true
I20240503 02:12:39.451892 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5920bfd139224d908c23f21d6a65308a" } modify_external_catalogs: true
I20240503 02:12:39.452733 5103 ts_tablet_manager.cc:1398] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.055s user 0.004s sys 0.002s
I20240503 02:12:39.453884 5103 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.305827 4672 raft_consensus.cc:2454] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.457623 5159 ts_tablet_manager.cc:1888] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.457734 5159 log.cc:1196] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2152f205cf1b4ec7ae03605bbfa007e1
I20240503 02:12:39.458117 5159 ts_tablet_manager.cc:1909] T 2152f205cf1b4ec7ae03605bbfa007e1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.461737 5103 tablet_bootstrap.cc:654] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.463680 5114 ts_tablet_manager.cc:1888] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.464831 5114 log.cc:1196] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:39.465378 5114 ts_tablet_manager.cc:1909] T e94031465ce7407dab0f90ad8809ac21 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:39.466363 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
I20240503 02:12:39.467721 4775 tablet_service.cc:1467] Processing CreateTablet for tablet a83d3bf41e8044e7a8f539429e43cdf3 (DEFAULT_TABLE table=default.Table_76c5d3de08164c7c893ba2fe01b1e38b [id=30b54a8aa5aa4241a076c3526d985e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.467909 4644 tablet_service.cc:1467] Processing CreateTablet for tablet a83d3bf41e8044e7a8f539429e43cdf3 (DEFAULT_TABLE table=default.Table_76c5d3de08164c7c893ba2fe01b1e38b [id=30b54a8aa5aa4241a076c3526d985e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.468287 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83d3bf41e8044e7a8f539429e43cdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.468405 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83d3bf41e8044e7a8f539429e43cdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.468427 5103 tablet_bootstrap.cc:492] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.469393 6317 raft_consensus.cc:482] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.469534 6317 raft_consensus.cc:504] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.470014 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716c4807e004760b34c0492cb7f522d with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.470149 6317 leader_election.cc:288] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.470686 6317 raft_consensus.cc:482] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:39.470798 6317 raft_consensus.cc:504] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.471282 6317 leader_election.cc:288] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.471735 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4 (DEFAULT_TABLE table=default.Table_1afe5e17ba0045cfb7cc930692d02e2d [id=8dcde7c990f54681a5fb019db77718a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.472043 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716c4807e004760b34c0492cb7f522d with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.472147 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.472620 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.472859 4808 raft_consensus.cc:2454] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:39.473240 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5e534255f170416690c18ca6bd571117
I20240503 02:12:39.473316 4608 leader_election.cc:302] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:39.473598 6317 raft_consensus.cc:2781] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.473670 6317 raft_consensus.cc:482] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.473724 6317 raft_consensus.cc:3037] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.475431 4274 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_b6a18c634a384042bb2ca1992f701886 [id=fbc72fd4039b499bbce796cad96e1be2] alter complete (version 2)
I20240503 02:12:39.475461 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f20e573d70bd46aab33466ccfc801465
I20240503 02:12:39.476213 6317 raft_consensus.cc:504] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:39.477365 4608 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 8750b02c5936412db525d2d3e77befeb
I20240503 02:12:39.477809 6317 leader_election.cc:288] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.477842 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.478014 4810 raft_consensus.cc:3037] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.478268 5103 ts_tablet_manager.cc:1367] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240503 02:12:39.479372 5103 raft_consensus.cc:348] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.481079 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716c4807e004760b34c0492cb7f522d with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:39.481479 4608 leader_election.cc:341] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 8750b02c5936412db525d2d3e77befeb
I20240503 02:12:39.481578 4608 leader_election.cc:302] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.482137 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4 (DEFAULT_TABLE table=default.Table_1afe5e17ba0045cfb7cc930692d02e2d [id=8dcde7c990f54681a5fb019db77718a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.482528 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.483362 6317 raft_consensus.cc:2726] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:39.480204 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.480643 4520 raft_consensus.cc:3037] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.485524 5103 raft_consensus.cc:374] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.485754 5103 raft_consensus.cc:729] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.486163 5103 consensus_queue.cc:260] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.487141 5103 ts_tablet_manager.cc:1398] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:39.481082 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f146f7cce0b24d3087bfe0973e2b7ef0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:39.485530 4529 raft_consensus.cc:2454] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:39.489508 6317 consensus_queue.cc:1035] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.490963 6296 raft_consensus.cc:482] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.491096 6296 raft_consensus.cc:504] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.491716 6296 leader_election.cc:288] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.492489 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.492653 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:39.493032 4746 leader_election.cc:341] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:39.493270 4746 leader_election.cc:341] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.493345 4746 leader_election.cc:302] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.318230 5115 ts_tablet_manager.cc:1888] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.493764 6349 raft_consensus.cc:2726] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.493855 5115 log.cc:1196] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/ea4dac17dca3442d8934ac357779b78a
I20240503 02:12:39.494221 5115 ts_tablet_manager.cc:1909] T ea4dac17dca3442d8934ac357779b78a P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.494576 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 640) took 3807 ms. Trace:
I20240503 02:12:39.494681 5115 rpcz_store.cc:276] 0503 02:12:35.687017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.687106 (+ 89us) service_pool.cc:225] Handling call
0503 02:12:39.294462 (+3607356us) ts_tablet_manager.cc:1120] Deleting tablet ea4dac17dca3442d8934ac357779b78a
0503 02:12:39.294470 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.307450 (+ 12980us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.318113 (+ 10663us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.494558 (+176445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3607050}
I20240503 02:12:39.495194 5115 tablet_replica.cc:330] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.495421 5115 raft_consensus.cc:2229] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.495604 5115 raft_consensus.cc:2258] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.496186 5115 ts_tablet_manager.cc:1875] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:39.508153 4608 consensus_peers.cc:485] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b91abaf51abb4c2ca65412c3771c1ef2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:39.515480 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e534255f170416690c18ca6bd571117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.515803 5105 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.515745 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_af6784c632dc44038b2633d2a6ccd1cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.515954 5105 ts_tablet_manager.cc:1367] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.015s
W20240503 02:12:39.516341 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af6784c632dc44038b2633d2a6ccd1cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.516713 5105 raft_consensus.cc:348] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.517184 5105 raft_consensus.cc:374] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.517247 5105 raft_consensus.cc:729] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.519096 5105 consensus_queue.cc:260] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.519779 5105 ts_tablet_manager.cc:1398] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:39.520222 6393 consensus_queue.cc:1035] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.520033 5105 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.521740 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5693354898d34e2285341bff83a126fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.522660 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5693354898d34e2285341bff83a126fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.523772 5105 tablet_bootstrap.cc:654] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.515219 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c012528450084a4498e1aafcccea12c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.525344 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_83898e39e7794bc582e22bab8a8e0e12" } new_table_name: "default.Table_1bda6b4dcbff43fc9555244a5fa395f8" modify_external_catalogs: true
W20240503 02:12:39.526796 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c012528450084a4498e1aafcccea12c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.531049 5868 raft_consensus.cc:686] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:39.531517 5868 consensus_queue.cc:237] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.532506 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ba2656ababd74dbeb917e32512185c57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.533156 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba2656ababd74dbeb917e32512185c57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.534780 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e534255f170416690c18ca6bd571117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.536744 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet f20e573d70bd46aab33466ccfc801465 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.537600 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet f20e573d70bd46aab33466ccfc801465 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.544185 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet f20e573d70bd46aab33466ccfc801465 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.544437 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 709) took 1680 ms. Trace:
I20240503 02:12:39.544467 6448 raft_consensus.cc:482] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.544531 5159 rpcz_store.cc:276] 0503 02:12:37.864587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.864690 (+ 103us) service_pool.cc:225] Handling call
0503 02:12:39.429540 (+1564850us) ts_tablet_manager.cc:1120] Deleting tablet 2152f205cf1b4ec7ae03605bbfa007e1
0503 02:12:39.429548 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.450754 (+ 21206us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.457529 (+ 6775us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.544409 (+ 86880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1896,"tablet-delete.queue_time_us":1564595}
I20240503 02:12:39.544829 5159 tablet_replica.cc:330] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.544680 6448 raft_consensus.cc:504] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.545012 5159 raft_consensus.cc:2229] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:39.545280 5159 raft_consensus.cc:2258] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.545783 5159 ts_tablet_manager.cc:1875] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.546087 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.546296 4672 raft_consensus.cc:2454] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.546802 4462 leader_election.cc:302] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.551790 5159 ts_tablet_manager.cc:1888] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.551968 5159 log.cc:1196] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/29e5c40f1f6b40308eed26b64d047494
I20240503 02:12:39.552412 5159 ts_tablet_manager.cc:1909] T 29e5c40f1f6b40308eed26b64d047494 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.552744 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 710) took 1687 ms. Trace:
I20240503 02:12:39.552882 5159 rpcz_store.cc:276] 0503 02:12:37.865284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.865498 (+ 214us) service_pool.cc:225] Handling call
0503 02:12:39.544788 (+1679290us) ts_tablet_manager.cc:1120] Deleting tablet 29e5c40f1f6b40308eed26b64d047494
0503 02:12:39.544796 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.548685 (+ 3889us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.551695 (+ 3010us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.552726 (+ 1031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679090}
I20240503 02:12:39.553373 5159 tablet_replica.cc:330] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.553596 5159 raft_consensus.cc:2229] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.553762 5159 raft_consensus.cc:2258] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.554349 5159 ts_tablet_manager.cc:1875] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.554675 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4 (DEFAULT_TABLE table=default.Table_1afe5e17ba0045cfb7cc930692d02e2d [id=8dcde7c990f54681a5fb019db77718a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.555989 4496 tablet_service.cc:1467] Processing CreateTablet for tablet cbaa94834cdc44eabe71242a698e3639 (DEFAULT_TABLE table=default.Table_b70b33ab1fd34982a6f9463002af9bac [id=44d868dbf2d74a5ab0661020e0b31627]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.557233 4810 raft_consensus.cc:2454] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:39.557334 4644 tablet_service.cc:1467] Processing CreateTablet for tablet cbaa94834cdc44eabe71242a698e3639 (DEFAULT_TABLE table=default.Table_b70b33ab1fd34982a6f9463002af9bac [id=44d868dbf2d74a5ab0661020e0b31627]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.558140 4608 leader_election.cc:302] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.558473 6317 raft_consensus.cc:2781] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.558723 6317 raft_consensus.cc:686] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:39.559090 5105 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.558948 6317 consensus_queue.cc:237] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.559202 5105 ts_tablet_manager.cc:1367] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240503 02:12:39.560734 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet ea4dac17dca3442d8934ac357779b78a (table default.Table_55c37198f9d2480784f643011b70dee8 [id=89060d0da21a4cf8b3d3c1ab1a8a3749]) successfully deleted
I20240503 02:12:39.559834 4781 tablet_service.cc:1467] Processing CreateTablet for tablet cbaa94834cdc44eabe71242a698e3639 (DEFAULT_TABLE table=default.Table_b70b33ab1fd34982a6f9463002af9bac [id=44d868dbf2d74a5ab0661020e0b31627]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.562057 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaa94834cdc44eabe71242a698e3639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.567018 5105 raft_consensus.cc:348] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.567886 5105 raft_consensus.cc:374] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.567961 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.569053 4810 raft_consensus.cc:2454] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.569155 5105 raft_consensus.cc:729] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.569761 5105 consensus_queue.cc:260] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.567723 6448 leader_election.cc:288] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:39.570886 5103 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.571388 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 54f03f7c9ae34104a606c81595725ce8 (DEFAULT_TABLE table=default.Table_34db9afaf7fc4588903d0918bcbb98a3 [id=56b2ab5e71c94159af626e9ec4066610]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.572291 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f03f7c9ae34104a606c81595725ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.572240 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 54f03f7c9ae34104a606c81595725ce8 (DEFAULT_TABLE table=default.Table_34db9afaf7fc4588903d0918bcbb98a3 [id=56b2ab5e71c94159af626e9ec4066610]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.572865 5103 tablet_bootstrap.cc:654] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.574095 5105 ts_tablet_manager.cc:1398] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240503 02:12:39.574405 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 27876b4d3eae40fab8c4b14370b76673 (DEFAULT_TABLE table=default.Table_a5d6a733e83e43b39bf0c003e569df5c [id=e9781cb7a3dd49028c7798f0066d1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.575378 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 27876b4d3eae40fab8c4b14370b76673 (DEFAULT_TABLE table=default.Table_a5d6a733e83e43b39bf0c003e569df5c [id=e9781cb7a3dd49028c7798f0066d1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.575718 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27876b4d3eae40fab8c4b14370b76673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.575747 5105 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.576941 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 27876b4d3eae40fab8c4b14370b76673 (DEFAULT_TABLE table=default.Table_a5d6a733e83e43b39bf0c003e569df5c [id=e9781cb7a3dd49028c7798f0066d1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.578708 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 2152f205cf1b4ec7ae03605bbfa007e1 (table default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]) successfully deleted
I20240503 02:12:39.583678 5159 ts_tablet_manager.cc:1888] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.586165 5159 log.cc:1196] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:39.587155 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaa94834cdc44eabe71242a698e3639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.587399 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27876b4d3eae40fab8c4b14370b76673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.587651 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f03f7c9ae34104a606c81595725ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.591647 6354 raft_consensus.cc:482] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.591779 6354 raft_consensus.cc:504] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.592370 6354 leader_election.cc:288] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.594787 5159 ts_tablet_manager.cc:1909] T ea7afb429d64497abcd9f1d0910651c3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.596145 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4225c1bc133d4e18a4a47fd58820ca10" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.586879 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 29e5c40f1f6b40308eed26b64d047494 (table default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]) successfully deleted
I20240503 02:12:39.598280 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a8d12a1274ba2b0f8c5d6b0e3d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:39.596745 4746 leader_election.cc:341] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.599062 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 711) took 1733 ms. Trace:
I20240503 02:12:39.599211 5159 rpcz_store.cc:276] 0503 02:12:37.865995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.866053 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:39.553328 (+1687275us) ts_tablet_manager.cc:1120] Deleting tablet ea7afb429d64497abcd9f1d0910651c3
0503 02:12:39.553338 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.565070 (+ 11732us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.583128 (+ 18058us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.599044 (+ 15916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687109}
I20240503 02:12:39.599165 5105 tablet_bootstrap.cc:654] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.601770 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 712) took 1735 ms. Trace:
I20240503 02:12:39.601892 5159 rpcz_store.cc:276] 0503 02:12:37.886271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.886579 (+ 308us) service_pool.cc:225] Handling call
0503 02:12:39.601676 (+1715097us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:39.601685 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.601754 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714867}
I20240503 02:12:39.603477 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4225c1bc133d4e18a4a47fd58820ca10" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.604001 4746 leader_election.cc:341] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.609037 4746 leader_election.cc:302] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.609506 6354 raft_consensus.cc:2726] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.609987 5159 tablet_replica.cc:330] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.610203 5159 raft_consensus.cc:2229] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.610411 4273 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d93e0ae90b5444228800872b90d94581
I20240503 02:12:39.610620 5159 raft_consensus.cc:2258] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.612171 5159 ts_tablet_manager.cc:1875] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:39.614980 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9c8547443694ce4b4b8018f190ac49f consensus state has no leader
I20240503 02:12:39.615496 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f88ecd4e6c544f43881964c3c4c128d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.616111 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f88ecd4e6c544f43881964c3c4c128d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.616613 5105 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.616760 5105 ts_tablet_manager.cc:1367] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20240503 02:12:39.617647 5105 raft_consensus.cc:348] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.617893 5105 raft_consensus.cc:374] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.617966 5105 raft_consensus.cc:729] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.618191 5105 consensus_queue.cc:260] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.618880 5105 ts_tablet_manager.cc:1398] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:39.619194 5105 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.620786 5105 tablet_bootstrap.cc:654] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.621572 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ea7afb429d64497abcd9f1d0910651c3 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
W20240503 02:12:39.623631 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8750b02c5936412db525d2d3e77befeb because the tablet was not found. No further retry: Not found: Tablet not found: 8750b02c5936412db525d2d3e77befeb
I20240503 02:12:39.627087 5159 ts_tablet_manager.cc:1888] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.627241 5159 log.cc:1196] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7394c9d374404fb199ab6ff170d7febe
I20240503 02:12:39.627605 5159 ts_tablet_manager.cc:1909] T 7394c9d374404fb199ab6ff170d7febe P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.628634 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 713) took 1741 ms. Trace:
I20240503 02:12:39.631126 5159 rpcz_store.cc:276] 0503 02:12:37.888148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.891348 (+ 3200us) service_pool.cc:225] Handling call
0503 02:12:39.609933 (+1718585us) ts_tablet_manager.cc:1120] Deleting tablet 7394c9d374404fb199ab6ff170d7febe
0503 02:12:39.609944 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.622009 (+ 12065us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.626994 (+ 4985us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.628616 (+ 1622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1718326}
I20240503 02:12:39.631850 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7394c9d374404fb199ab6ff170d7febe (table default.Table_be53cf9a5c3a418d9dc24120c17d3a4b [id=646347328e89442b8713fa69f5078052]) successfully deleted
I20240503 02:12:39.633071 5159 tablet_replica.cc:330] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.636492 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_2df5b1aa30a0494f86117be55903a129 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 2 got 0
I20240503 02:12:39.636611 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_887df63808734e458d658273fc7d2fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.636839 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:39.637060 5159 raft_consensus.cc:2229] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:39.637212 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_887df63808734e458d658273fc7d2fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.637398 5159 raft_consensus.cc:2258] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.637372 4273 catalog_manager.cc:5484] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:39.637840 4273 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:39.638479 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7461aa76a1af41cb9253285bda708f30
I20240503 02:12:39.638779 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c0df938d7e494f96a93c3c23baa3bb09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.639168 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b839b2ab5a8a4a399e63762bec827155" } new_table_name: "default.Table_958b53896e2945d7a5fc23d65f0dc378" modify_external_catalogs: true
W20240503 02:12:39.639345 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0df938d7e494f96a93c3c23baa3bb09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.640323 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea1eb32394b04b72258d24196202b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:39.640545 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fa25fee059b46ec915bcc1a01cb8b27 consensus state has no leader
I20240503 02:12:39.640985 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet b161628065704c798020edbe098844c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.640899 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a2518b080992412aa7ae7abdd643bdf7" } new_table_name: "default.Table_564e48fcfdc54bd2b6352595749f7e78" modify_external_catalogs: true
I20240503 02:12:39.641229 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 7461aa76a1af41cb9253285bda708f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.641448 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.641872 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.642313 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8" } modify_external_catalogs: true
W20240503 02:12:39.642352 4274 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:37265 (request call id 102) took 1162 ms (client timeout 1000 ms). Trace:
I20240503 02:12:39.642537 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:39.642517 4274 rpcz_store.cc:269] 0503 02:12:38.479798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.503534 (+ 23736us) service_pool.cc:225] Handling call
0503 02:12:38.507672 (+ 4138us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x616000021d54. stack: 00007fd3e71be756 00007fd3e71be510 00007fd3e61f148d 0000000000955804 00000000009557cd 00007fd3f5cfe15d 00007fd3f5cfe253 00007fd3f2a2fd41 00007fd3fa61c34d 00007fd3fa7a3668 00007fd3f1797786 00007fd3f17974b5 00007fd3edec604e 00007fd3edec4f2e 00007fd3edecb405 00007fd3edecbd2f
0503 02:12:39.642309 (+1134637us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:38.715257 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:38.715510 (+ 253us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:38.715517 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:38.715522 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:38.715798 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:38.715839 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:38.715848 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:38.717022 (+ 1174us) tablet.cc:699] Row locks acquired
0503 02:12:38.717026 (+ 4us) write_op.cc:260] PREPARE: finished
0503 02:12:38.717092 (+ 66us) write_op.cc:270] Start()
0503 02:12:38.717165 (+ 73us) write_op.cc:276] Timestamp: P: 1714702358717049 usec, L: 0
0503 02:12:38.717170 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 02:12:38.717466 (+ 296us) log.cc:842] Serialized 901 byte log entry
0503 02:12:39.129976 (+412510us) op_driver.cc:444] REPLICATION: finished
0503 02:12:39.130762 (+ 786us) write_op.cc:301] APPLY: starting
0503 02:12:39.130775 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:39.130804 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:39.130833 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:39.130891 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:39.130918 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:39.130925 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:39.130980 (+ 55us) log.cc:842] Serialized 29 byte log entry
0503 02:12:39.131019 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.131035 (+ 16us) write_op.cc:454] Released schema lock
0503 02:12:39.131041 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:39.131090 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.131094 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 02:12:39.442991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:39.443109 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:39.443117 (+ 8us) write_op.cc:435] Acquired schema lock
0503 02:12:39.443119 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:39.443252 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:39.443265 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:39.443269 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:39.443337 (+ 68us) tablet.cc:699] Row locks acquired
0503 02:12:39.443340 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:39.443379 (+ 39us) write_op.cc:270] Start()
0503 02:12:39.443418 (+ 39us) write_op.cc:276] Timestamp: P: 1714702359443314 usec, L: 0
0503 02:12:39.443422 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:39.443601 (+ 179us) log.cc:842] Serialized 356 byte log entry
0503 02:12:39.474402 (+ 30801us) op_driver.cc:444] REPLICATION: finished
0503 02:12:39.474550 (+ 148us) write_op.cc:301] APPLY: starting
0503 02:12:39.474561 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:39.474587 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:39.474617 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:39.474652 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:39.474678 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:39.474685 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:39.474756 (+ 71us) log.cc:842] Serialized 23 byte log entry
0503 02:12:39.474791 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.474803 (+ 12us) write_op.cc:454] Released schema lock
0503 02:12:39.474809 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:39.474852 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:39.474856 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20,"reported_tablets":39,"spinlock_wait_cycles":5872384,"child_traces":[["op",{"apply.queue_time_us":741,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":380,"mutex_wait_us":24135,"num_ops":2,"prepare.queue_time_us":24323,"prepare.run_cpu_time_us":1131,"prepare.run_wall_time_us":2663,"replication_time_us":412793}],["op",{"apply.queue_time_us":120,"apply.run_cpu_time_us":344,"apply.run_wall_time_us":344,"num_ops":1,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":686,"replication_time_us":30968}]]}
I20240503 02:12:39.643828 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 7461aa76a1af41cb9253285bda708f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.644074 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.644120 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f9994ca5aea48d3bc25c097dddba3c3) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.644281 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7461aa76a1af41cb9253285bda708f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.645705 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.645896 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 59855ab3e42b4e019b769007ccbbc360 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.647243 5105 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.647397 5105 ts_tablet_manager.cc:1367] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240503 02:12:39.648167 5105 raft_consensus.cc:348] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.648443 5105 raft_consensus.cc:374] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.648514 5105 raft_consensus.cc:729] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.642514 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8" } modify_external_catalogs: true
I20240503 02:12:39.648758 5105 consensus_queue.cc:260] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.649435 5105 ts_tablet_manager.cc:1398] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:39.649710 5105 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.651058 5105 tablet_bootstrap.cc:654] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.570569 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 54f03f7c9ae34104a606c81595725ce8 (DEFAULT_TABLE table=default.Table_34db9afaf7fc4588903d0918bcbb98a3 [id=56b2ab5e71c94159af626e9ec4066610]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.570099 6478 raft_consensus.cc:2781] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.653136 6478 raft_consensus.cc:482] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.653231 6478 raft_consensus.cc:3037] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.644490 5159 ts_tablet_manager.cc:1875] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.652029 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178760c21904207b6c8317093871f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.652163 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.652277 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b962c9264f87b798df33544cb15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.652709 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.654476 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b72c20a4b5a4d2ea266f031733d1aab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.654750 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.655230 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 53017352b8f34955b116bbfd32f04854 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.655683 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 885129a4efc5475b97b7ac68e1db5e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.657213 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5faf42ddb1754ee3b8a4f879de32cd43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.658457 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152f205cf1b4ec7ae03605bbfa007e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.658875 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56701ed6424c60bf1a985eea1936fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.659299 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5c40f1f6b40308eed26b64d047494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.659811 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ef3c6e5e7d40249b112144975ba220 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.664086 6354 raft_consensus.cc:482] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.664214 6354 raft_consensus.cc:504] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.665047 6354 leader_election.cc:288] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:39.664703 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5faf42ddb1754ee3b8a4f879de32cd43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.665788 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.665788 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:39.666342 4746 leader_election.cc:341] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:39.666574 4746 leader_election.cc:341] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.666636 4746 leader_election.cc:302] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.666996 6354 raft_consensus.cc:2726] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.667521 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 396b48865e954d05b9ba8084a2e7f29b with delete_type TABLET_DATA_DELETED (replaced by tablet 84b6e303e34a465caffed6c9e1e6bf74) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.668033 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4dac17dca3442d8934ac357779b78a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.668459 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.668876 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet e94031465ce7407dab0f90ad8809ac21 with delete_type TABLET_DATA_DELETED (replaced by tablet a88710ac9062472f80bf6fcfb82793a0) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.671133 5105 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.671288 5105 ts_tablet_manager.cc:1367] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240503 02:12:39.672060 5105 raft_consensus.cc:348] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.672237 5105 raft_consensus.cc:374] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.672291 5105 raft_consensus.cc:729] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.672483 5105 consensus_queue.cc:260] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.673013 5105 ts_tablet_manager.cc:1398] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:39.673274 5105 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.674500 5105 tablet_bootstrap.cc:654] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.675186 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9b6b0e4d8d9f43bf9322382e80b8e842" } modify_external_catalogs: true
I20240503 02:12:39.675401 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9b6b0e4d8d9f43bf9322382e80b8e842" } modify_external_catalogs: true
I20240503 02:12:39.678618 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b48ee0fa4b364d7ca10d28cc8224840e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.682487 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b48ee0fa4b364d7ca10d28cc8224840e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.684690 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e534255f170416690c18ca6bd571117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.685011 6296 raft_consensus.cc:482] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.685313 6296 raft_consensus.cc:504] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.686275 6296 leader_election.cc:288] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.686977 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43119471f91c4a1ba900f267c88db9a6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.687251 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43119471f91c4a1ba900f267c88db9a6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.687860 4746 leader_election.cc:341] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:39.688082 4746 leader_election.cc:341] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.688167 4746 leader_election.cc:302] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.688937 6349 raft_consensus.cc:2726] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.696763 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9fa3df92b6e544de9ca76d98706eebec
I20240503 02:12:39.697520 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4284eccc6a0f475bbfe7d8a92029b257"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.698117 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4284eccc6a0f475bbfe7d8a92029b257 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.699728 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3cc52a664da0462f93bc3453acd53989" } new_table_name: "default.Table_aab4f07f807f43f291c6bc552189e280" modify_external_catalogs: true
I20240503 02:12:39.700553 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa3df92b6e544de9ca76d98706eebec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:39.704857 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe35f00dd4e74dc8b6b61eefe2bafaab consensus state has no leader
I20240503 02:12:39.705217 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf5d0a99df5945038f6a923ebeae4b32" } modify_external_catalogs: true
I20240503 02:12:39.705399 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf5d0a99df5945038f6a923ebeae4b32" } modify_external_catalogs: true
I20240503 02:12:39.706166 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa3df92b6e544de9ca76d98706eebec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.703198 5159 ts_tablet_manager.cc:1888] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.706502 5159 log.cc:1196] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/f146f7cce0b24d3087bfe0973e2b7ef0
I20240503 02:12:39.709408 5159 ts_tablet_manager.cc:1909] T f146f7cce0b24d3087bfe0973e2b7ef0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.709951 5105 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.710093 5105 ts_tablet_manager.cc:1367] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.004s
I20240503 02:12:39.710528 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 714) took 1818 ms. Trace:
I20240503 02:12:39.710635 5159 rpcz_store.cc:276] 0503 02:12:37.896669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.898544 (+ 1875us) service_pool.cc:225] Handling call
0503 02:12:39.633007 (+1734463us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:39.633020 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.690781 (+ 57761us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.703097 (+ 12316us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.710511 (+ 7414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1734228}
I20240503 02:12:39.710810 5105 raft_consensus.cc:348] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.711282 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet f146f7cce0b24d3087bfe0973e2b7ef0 (table default.Table_18dc31f244e94d6ab3c576eaae5e408b [id=64f4026243214e1bbed391c6e0e7df76]) successfully deleted
I20240503 02:12:39.711341 5105 raft_consensus.cc:374] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.712280 5105 raft_consensus.cc:729] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.711659 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 715) took 1812 ms. Trace:
W20240503 02:12:39.712626 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf10c48d165047668fbd5ca17f917fe3 consensus state has no leader
I20240503 02:12:39.712587 5105 consensus_queue.cc:260] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.712641 5159 rpcz_store.cc:276] 0503 02:12:37.898993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.901809 (+ 2816us) service_pool.cc:225] Handling call
0503 02:12:39.711568 (+1809759us) ts_tablet_manager.cc:1120] Deleting tablet 19fb0929bd234788b5313d96d2e4845c
0503 02:12:39.711577 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.711643 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809580}
I20240503 02:12:39.713129 5105 ts_tablet_manager.cc:1398] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:39.713374 6296 raft_consensus.cc:482] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.713660 6296 raft_consensus.cc:504] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.713850 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:39.713812 5105 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:39.714354 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 19fb0929bd234788b5313d96d2e4845c because the tablet was not found. No further retry: Not found: Tablet not found: 19fb0929bd234788b5313d96d2e4845c
I20240503 02:12:39.714979 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.715471 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:39.718673 4274 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:39.714242 6296 leader_election.cc:288] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.719614 4274 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:39.714735 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 716) took 1812 ms. Trace:
W20240503 02:12:39.719141 4746 leader_election.cc:341] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.720165 5159 rpcz_store.cc:276] 0503 02:12:37.902807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.902855 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:39.714645 (+1811790us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:39.714654 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.714718 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1811633}
W20240503 02:12:39.720304 4746 leader_election.cc:341] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.720383 4746 leader_election.cc:302] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:39.720608 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet e94031465ce7407dab0f90ad8809ac21 because the tablet was not found. No further retry: Not found: Tablet not found: e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:39.720785 6296 raft_consensus.cc:2726] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.721036 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 721) took 1639 ms. Trace:
I20240503 02:12:39.721136 5159 rpcz_store.cc:276] 0503 02:12:38.081804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.081846 (+ 42us) service_pool.cc:225] Handling call
0503 02:12:39.720837 (+1638991us) ts_tablet_manager.cc:1120] Deleting tablet 8f8b6599206549dd8ddf449a52c61fce
0503 02:12:39.720847 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.721022 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638783}
I20240503 02:12:39.721609 5105 tablet_bootstrap.cc:654] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:39.721611 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8f8b6599206549dd8ddf449a52c61fce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8b6599206549dd8ddf449a52c61fce already in progress: creating tablet
I20240503 02:12:39.721947 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 720) took 1640 ms. Trace:
I20240503 02:12:39.725987 5159 rpcz_store.cc:276] 0503 02:12:38.081671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.085101 (+ 3430us) service_pool.cc:225] Handling call
0503 02:12:39.721854 (+1636753us) ts_tablet_manager.cc:1120] Deleting tablet 0e89c1cf6ed54d2fafb2173ea9373ab5
0503 02:12:39.721862 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.721932 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636564}
W20240503 02:12:39.726507 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 already in progress: creating tablet
I20240503 02:12:39.741037 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa3df92b6e544de9ca76d98706eebec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.745518 5106 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.745738 5106 ts_tablet_manager.cc:1367] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.372s user 0.012s sys 0.025s
W20240503 02:12:39.746116 6491 log.cc:924] Time spent T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240503 02:12:39.746572 5106 raft_consensus.cc:348] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.746762 5106 raft_consensus.cc:374] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.746814 5106 raft_consensus.cc:729] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.747004 5106 consensus_queue.cc:260] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.748873 5106 ts_tablet_manager.cc:1398] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:39.749679 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaa94834cdc44eabe71242a698e3639. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.750656 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 626) took 4635 ms. Trace:
I20240503 02:12:39.750746 5114 rpcz_store.cc:276] 0503 02:12:35.115719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.115781 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:39.396987 (+4281206us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:39.396996 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.440266 (+ 43270us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.463590 (+ 23324us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.750635 (+287045us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103606,"tablet-delete.queue_time_us":4280946}
I20240503 02:12:39.751144 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 627) took 4482 ms. Trace:
I20240503 02:12:39.751209 5114 rpcz_store.cc:276] 0503 02:12:35.268580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.268635 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:39.751026 (+4482391us) ts_tablet_manager.cc:1120] Deleting tablet 6ecdb634fb1c4337b2eae0c844fbb1dc
0503 02:12:39.751034 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.751129 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4482149}
I20240503 02:12:39.751493 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 628) took 4463 ms. Trace:
I20240503 02:12:39.751550 5114 rpcz_store.cc:276] 0503 02:12:35.287923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.287972 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:39.751392 (+4463420us) ts_tablet_manager.cc:1120] Deleting tablet ee1fac803dd84eee8722f0a14559faf6
0503 02:12:39.751399 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.751481 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4463179}
I20240503 02:12:39.751823 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 629) took 4448 ms. Trace:
I20240503 02:12:39.751891 5114 rpcz_store.cc:276] 0503 02:12:35.303541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.303603 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:39.751739 (+4448136us) ts_tablet_manager.cc:1120] Deleting tablet 9290a61c21fa456eb3c4f9b0dbebc864
0503 02:12:39.751747 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.751811 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4383116}
I20240503 02:12:39.752139 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 641) took 4092 ms. Trace:
I20240503 02:12:39.752197 5114 rpcz_store.cc:276] 0503 02:12:35.659639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.659696 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:39.752065 (+4092369us) ts_tablet_manager.cc:1120] Deleting tablet 8c6c80efe35343989f46a2c181f38129
0503 02:12:39.752072 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.752129 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090644}
I20240503 02:12:39.752435 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 642) took 4092 ms. Trace:
I20240503 02:12:39.752487 5114 rpcz_store.cc:276] 0503 02:12:35.660038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.660125 (+ 87us) service_pool.cc:225] Handling call
0503 02:12:39.752356 (+4092231us) ts_tablet_manager.cc:1120] Deleting tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2
0503 02:12:39.752363 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.752417 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090598}
I20240503 02:12:39.752737 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 643) took 4092 ms. Trace:
I20240503 02:12:39.752794 5114 rpcz_store.cc:276] 0503 02:12:35.660428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.660478 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:39.752648 (+4092170us) ts_tablet_manager.cc:1120] Deleting tablet 3e94a9c61cb74cc08ca5867ae85ecdf0
0503 02:12:39.752657 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.752724 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4041718}
I20240503 02:12:39.753018 5114 tablet_replica.cc:330] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:39.753180 5114 raft_consensus.cc:2229] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.753526 5114 raft_consensus.cc:2258] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.753651 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27876b4d3eae40fab8c4b14370b76673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.754021 5114 ts_tablet_manager.cc:1875] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.755936 5103 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.756049 5103 ts_tablet_manager.cc:1367] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.185s user 0.007s sys 0.002s
I20240503 02:12:39.757299 5103 raft_consensus.cc:348] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.757491 5103 raft_consensus.cc:374] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.757541 5103 raft_consensus.cc:729] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.757604 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f03f7c9ae34104a606c81595725ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.757717 5103 consensus_queue.cc:260] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.758183 5103 ts_tablet_manager.cc:1398] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:39.758450 5103 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.760118 5103 tablet_bootstrap.cc:654] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.761059 5114 ts_tablet_manager.cc:1888] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.761859 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet e94031465ce7407dab0f90ad8809ac21 (table default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]) successfully deleted
W20240503 02:12:39.762192 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6ecdb634fb1c4337b2eae0c844fbb1dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecdb634fb1c4337b2eae0c844fbb1dc already in progress: creating tablet
W20240503 02:12:39.762462 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ee1fac803dd84eee8722f0a14559faf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1fac803dd84eee8722f0a14559faf6 already in progress: creating tablet
W20240503 02:12:39.762748 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9290a61c21fa456eb3c4f9b0dbebc864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9290a61c21fa456eb3c4f9b0dbebc864 already in progress: creating tablet
W20240503 02:12:39.762954 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8c6c80efe35343989f46a2c181f38129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6c80efe35343989f46a2c181f38129 already in progress: creating tablet
W20240503 02:12:39.763126 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ba5250ef83a4ddbafbc79e1ae21c9f2 already in progress: creating tablet
W20240503 02:12:39.763306 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e94a9c61cb74cc08ca5867ae85ecdf0 already in progress: creating tablet
I20240503 02:12:39.767966 5114 log.cc:1196] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:39.768342 5114 ts_tablet_manager.cc:1909] T 65774b69a8bd4664a6b55b84eba37f7f P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:39.771514 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 644) took 4110 ms. Trace:
I20240503 02:12:39.774874 5114 rpcz_store.cc:276] 0503 02:12:35.660702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.660753 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:39.752982 (+4092229us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:39.752989 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.758034 (+ 5045us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.760986 (+ 2952us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.771492 (+ 10506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4041988}
I20240503 02:12:39.776494 5106 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.776450 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 65774b69a8bd4664a6b55b84eba37f7f (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
I20240503 02:12:39.770107 6478 raft_consensus.cc:504] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.777593 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:39.777781 4667 raft_consensus.cc:3037] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.777854 5106 tablet_bootstrap.cc:654] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.781095 4667 raft_consensus.cc:2454] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.781638 4462 leader_election.cc:302] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.782050 6448 raft_consensus.cc:2781] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.782210 6448 raft_consensus.cc:686] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:39.782516 6448 consensus_queue.cc:237] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.784057 5114 tablet_replica.cc:330] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:39.784332 5114 raft_consensus.cc:2229] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.784610 5114 raft_consensus.cc:2258] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.785362 5114 ts_tablet_manager.cc:1875] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.789799 6478 leader_election.cc:288] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:39.789742 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d44fe8f6334572996665bdec60e2e5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.789945 4808 raft_consensus.cc:3037] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.792460 4808 raft_consensus.cc:2454] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.789762 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 723) took 1528 ms. Trace:
I20240503 02:12:39.794502 5159 rpcz_store.cc:276] 0503 02:12:38.261446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.261880 (+ 434us) service_pool.cc:225] Handling call
0503 02:12:39.726771 (+1464891us) ts_tablet_manager.cc:1120] Deleting tablet 56aa7303f7784d2289a8da9653d1e317
0503 02:12:39.789642 (+ 62871us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.789748 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464688}
I20240503 02:12:39.794879 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 724) took 1531 ms. Trace:
I20240503 02:12:39.794956 5159 rpcz_store.cc:276] 0503 02:12:38.263634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.263698 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:39.794765 (+1531067us) ts_tablet_manager.cc:1120] Deleting tablet 20b6a18be64049049e1a083bf4af51ba
0503 02:12:39.794776 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.794864 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530898}
I20240503 02:12:39.795209 5159 tablet_replica.cc:330] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.795408 5159 raft_consensus.cc:2229] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:39.795703 5159 raft_consensus.cc:2258] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.796236 5159 ts_tablet_manager.cc:1875] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:39.797165 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 56aa7303f7784d2289a8da9653d1e317 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56aa7303f7784d2289a8da9653d1e317 already in progress: creating tablet
W20240503 02:12:39.797466 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 20b6a18be64049049e1a083bf4af51ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b6a18be64049049e1a083bf4af51ba already in progress: creating tablet
W20240503 02:12:39.797673 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 372c128e62d04b7aa1bc3c07291d9876 has no consensus state
I20240503 02:12:39.804152 5159 ts_tablet_manager.cc:1888] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.804302 5159 log.cc:1196] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/da12dd28e03f462b905d1844e2ad3340
I20240503 02:12:39.804653 5159 ts_tablet_manager.cc:1909] T da12dd28e03f462b905d1844e2ad3340 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.805435 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 725) took 1293 ms. Trace:
I20240503 02:12:39.805552 5159 rpcz_store.cc:276] 0503 02:12:38.512022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.512083 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:39.795159 (+1283076us) ts_tablet_manager.cc:1120] Deleting tablet da12dd28e03f462b905d1844e2ad3340
0503 02:12:39.795167 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.800958 (+ 5791us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.804068 (+ 3110us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.805417 (+ 1349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282875}
I20240503 02:12:39.806015 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet da12dd28e03f462b905d1844e2ad3340 (table default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]) successfully deleted
I20240503 02:12:39.806355 5159 tablet_replica.cc:330] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.806567 5159 raft_consensus.cc:2229] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.806728 5159 raft_consensus.cc:2258] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.807296 5159 ts_tablet_manager.cc:1875] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.807304 5115 ts_tablet_manager.cc:1888] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.807410 5115 log.cc:1196] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/8750b02c5936412db525d2d3e77befeb
I20240503 02:12:39.807781 5115 ts_tablet_manager.cc:1909] T 8750b02c5936412db525d2d3e77befeb P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.808095 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 661) took 2737 ms. Trace:
I20240503 02:12:39.808307 5115 rpcz_store.cc:276] 0503 02:12:37.070413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.074437 (+ 4024us) service_pool.cc:225] Handling call
0503 02:12:39.495148 (+2420711us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:39.495157 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.688855 (+193698us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.807198 (+118343us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.808076 (+ 878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420466}
I20240503 02:12:39.808782 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 8750b02c5936412db525d2d3e77befeb (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]) successfully deleted
I20240503 02:12:39.809126 5115 tablet_replica.cc:330] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.809350 5115 raft_consensus.cc:2229] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:39.810387 5115 raft_consensus.cc:2258] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.811093 5115 ts_tablet_manager.cc:1875] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.814527 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8f8f3c9fd776473b8dd8eed18d12c897" } modify_external_catalogs: true
I20240503 02:12:39.814786 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8f8f3c9fd776473b8dd8eed18d12c897" } modify_external_catalogs: true
I20240503 02:12:39.820060 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_887df63808734e458d658273fc7d2fff" } new_table_name: "default.Table_196b58d9443a45dba25db7d08ee04d20" modify_external_catalogs: true
I20240503 02:12:39.826396 4667 raft_consensus.cc:1261] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:39.826961 6485 consensus_queue.cc:1035] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:39.834175 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c90bb18baae414d8d9a020cd38196fb has no consensus state
I20240503 02:12:39.837916 5103 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:39.838239 5103 ts_tablet_manager.cc:1367] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.005s
I20240503 02:12:39.839635 5103 raft_consensus.cc:348] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.839972 5103 raft_consensus.cc:374] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.840098 5103 raft_consensus.cc:729] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.840476 5103 consensus_queue.cc:260] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.841751 5103 ts_tablet_manager.cc:1398] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:39.842417 5103 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:39.845238 5103 tablet_bootstrap.cc:654] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.848723 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 426cbf5ddc5a4944a487b305d5895eae (DEFAULT_TABLE table=default.Table_3fabf651e344476ca18d32ef633dd619 [id=c7e62260bd7c4dfc8b52df645339dcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.849160 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426cbf5ddc5a4944a487b305d5895eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.849030 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 426cbf5ddc5a4944a487b305d5895eae (DEFAULT_TABLE table=default.Table_3fabf651e344476ca18d32ef633dd619 [id=c7e62260bd7c4dfc8b52df645339dcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.850898 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_92359e32d7a04637bfb894a7f9e1fb42 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 1 got 0
I20240503 02:12:39.851127 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 2c5d58f97b58481ea1dfa75397a32ae2 (table default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb]). Expected version 1 got 0
I20240503 02:12:39.851943 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:39.852085 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_2df5b1aa30a0494f86117be55903a129 [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 2 got 0
I20240503 02:12:39.856326 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426cbf5ddc5a4944a487b305d5895eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.856007 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_817b839a30804e609f4916cf44d4dafd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.858402 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_817b839a30804e609f4916cf44d4dafd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.856498 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_45a63336226e4bd29bf8fb7d9c8e02a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.859704 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45a63336226e4bd29bf8fb7d9c8e02a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.856845 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf5d681047e040968c0436492ac29ac6" } modify_external_catalogs: true
I20240503 02:12:39.860745 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf5d681047e040968c0436492ac29ac6" } modify_external_catalogs: true
I20240503 02:12:39.862027 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:39.862319 4273 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_918295a06e0d4e09af7c72203e0c3448 [id=713930e2953343799de0e9c465e1c083] alter complete (version 2)
I20240503 02:12:39.865375 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.865818 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_27706a397e30464e8687aacf693fc733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.866158 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_92359e32d7a04637bfb894a7f9e1fb42" } new_table_name: "default.Table_7e85ce8838ff4bb8a5551abe22389f43" modify_external_catalogs: true
W20240503 02:12:39.866492 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27706a397e30464e8687aacf693fc733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.866568 6482 raft_consensus.cc:482] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.866694 6482 raft_consensus.cc:504] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.867334 6482 leader_election.cc:288] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:39.868002 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.868206 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.868281 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.868360 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.869105 4811 raft_consensus.cc:1261] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Refusing update from remote peer f4f9f079230a468f90e5e3a58f254b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:12:39.870085 4462 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:39.870605 6448 consensus_queue.cc:1035] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.875330 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 44707b6439b141b7b63f49f34f25b37a
I20240503 02:12:39.878957 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 44707b6439b141b7b63f49f34f25b37a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.879442 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_38a612f7f6034cf28b8329d12c6f97ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.879724 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cf10c48d165047668fbd5ca17f917fe3 consensus state has no leader
W20240503 02:12:39.880059 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38a612f7f6034cf28b8329d12c6f97ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.883247 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ab093285c39b4ce7a0bb3de9d016a918
I20240503 02:12:39.883378 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 44707b6439b141b7b63f49f34f25b37a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.884395 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.884022 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:39.886834 4608 consensus_peers.cc:485] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2f9b962c9264f87b798df33544cb15d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:39.889209 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4fb59bbf7b68431c86d88f27ef2528b0
I20240503 02:12:39.889225 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394c9d374404fb199ab6ff170d7febe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.889451 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet b91abaf51abb4c2ca65412c3771c1ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.889699 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 65774b69a8bd4664a6b55b84eba37f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.889948 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d44fe8f6334572996665bdec60e2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.890173 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 44707b6439b141b7b63f49f34f25b37a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.890645 6183 raft_consensus.cc:482] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.890717 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.890799 6183 raft_consensus.cc:504] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.891381 6183 leader_election.cc:288] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:39.890195 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet ab093285c39b4ce7a0bb3de9d016a918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.891901 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9c84ec1ce10e42188f751afc07da15f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.890645 4462 leader_election.cc:341] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:39.892524 4462 leader_election.cc:302] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
W20240503 02:12:39.892665 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c84ec1ce10e42188f751afc07da15f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.893157 6448 raft_consensus.cc:2726] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.894073 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:39.894304 4811 raft_consensus.cc:2454] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:39.894497 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.895823 4462 leader_election.cc:302] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.895882 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b21a3010a4470288fbdd684ef64e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.895831 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.896174 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.896194 6448 raft_consensus.cc:2781] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.896312 6448 raft_consensus.cc:482] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.896378 6448 raft_consensus.cc:3037] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:39.897012 4462 consensus_peers.cc:485] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41ef3c6e5e7d40249b112144975ba220. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:39.897336 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.898396 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:39.899093 4462 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.899698 6448 raft_consensus.cc:504] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.901115 6448 leader_election.cc:288] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:39.901485 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.901753 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:39.901933 4808 raft_consensus.cc:3037] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.901906 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb59bbf7b68431c86d88f27ef2528b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.904410 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9ffe0b87ba144949b561f827d3d06611" } modify_external_catalogs: true
I20240503 02:12:39.904621 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9ffe0b87ba144949b561f827d3d06611" } modify_external_catalogs: true
I20240503 02:12:39.904703 6374 raft_consensus.cc:482] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:39.904832 6374 raft_consensus.cc:504] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:39.905556 6374 leader_election.cc:288] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.905756 5114 ts_tablet_manager.cc:1888] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.905858 5114 log.cc:1196] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/ea4dac17dca3442d8934ac357779b78a
I20240503 02:12:39.906257 5114 ts_tablet_manager.cc:1909] T ea4dac17dca3442d8934ac357779b78a P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:39.906566 4608 leader_election.cc:341] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 159ea1eb32394b04b72258d24196202b
I20240503 02:12:39.906426 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ffa48032226a4aa4859ee9f4a330014b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.907495 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffa48032226a4aa4859ee9f4a330014b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.909281 4808 raft_consensus.cc:2454] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:39.902554 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_055c133ec2d14ff7b5940ff3d79ad420"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:39.909536 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 645) took 4165 ms. Trace:
W20240503 02:12:39.909857 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_055c133ec2d14ff7b5940ff3d79ad420 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.909935 4462 leader_election.cc:302] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:39.910569 6448 raft_consensus.cc:2781] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.910841 6448 raft_consensus.cc:686] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
W20240503 02:12:39.911054 4608 leader_election.cc:341] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 159ea1eb32394b04b72258d24196202b
I20240503 02:12:39.911170 4608 leader_election.cc:302] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.911134 6448 consensus_queue.cc:237] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.911579 6374 raft_consensus.cc:2726] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:39.911993 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:39.912561 4462 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:39.915752 4170 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 90643.5us
I20240503 02:12:39.918998 5114 rpcz_store.cc:276] 0503 02:12:35.744446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:35.744513 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:39.784003 (+4039490us) ts_tablet_manager.cc:1120] Deleting tablet ea4dac17dca3442d8934ac357779b78a
0503 02:12:39.784014 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.813352 (+ 29338us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.905647 (+ 92295us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.909504 (+ 3857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4039193}
I20240503 02:12:39.919835 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet ea4dac17dca3442d8934ac357779b78a (table default.Table_55c37198f9d2480784f643011b70dee8 [id=89060d0da21a4cf8b3d3c1ab1a8a3749]) successfully deleted
I20240503 02:12:39.923599 6513 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:39.924613 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 667) took 2970 ms. Trace:
I20240503 02:12:39.924562 6510 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:39.924795 5114 rpcz_store.cc:276] 0503 02:12:36.954588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.954638 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:39.924388 (+2969750us) ts_tablet_manager.cc:1120] Deleting tablet 5cd296bb3b574d6cbb0252ad10ada00c
0503 02:12:39.924401 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.924595 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969557}
I20240503 02:12:39.925212 6511 tablet.cc:1722] T e366ce9d79b3458fb26e96b6e9278fbe P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 02:12:39.925427 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5cd296bb3b574d6cbb0252ad10ada00c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cd296bb3b574d6cbb0252ad10ada00c already in progress: creating tablet
I20240503 02:12:39.926081 5106 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:39.926294 5106 ts_tablet_manager.cc:1367] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.150s user 0.005s sys 0.005s
I20240503 02:12:39.926321 6516 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:39.927184 5106 raft_consensus.cc:348] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.927477 5106 raft_consensus.cc:374] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.927660 5106 raft_consensus.cc:729] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.928119 5106 consensus_queue.cc:260] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.928591 6515 tablet.cc:1722] T e366ce9d79b3458fb26e96b6e9278fbe P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:39.929198 5106 ts_tablet_manager.cc:1398] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:39.929625 5106 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:39.929808 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 668) took 2975 ms. Trace:
I20240503 02:12:39.934422 6296 raft_consensus.cc:482] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.934437 5114 rpcz_store.cc:276] 0503 02:12:36.954703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.954879 (+ 176us) service_pool.cc:225] Handling call
0503 02:12:39.929645 (+2974766us) ts_tablet_manager.cc:1120] Deleting tablet 6d08e7d17f1644749a9a2e7175b3191a
0503 02:12:39.929655 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.929787 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2974618}
I20240503 02:12:39.934569 6296 raft_consensus.cc:504] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:39.934968 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6d08e7d17f1644749a9a2e7175b3191a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d08e7d17f1644749a9a2e7175b3191a already in progress: creating tablet
I20240503 02:12:39.935218 6296 leader_election.cc:288] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:39.936132 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:39.939630 4746 leader_election.cc:341] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.943846 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_99049a1e00e84790bce035787060bd61" } new_table_name: "default.Table_aa8c3d142c5746cc83cb777a33f38060" modify_external_catalogs: true
I20240503 02:12:39.945138 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 41531ddcbf66465094d8f06e1116b3be
I20240503 02:12:39.945832 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_923e68dde96e47489191170eb93b19ab" } new_table_name: "default.Table_09f7e87cede74bea95028b5029888b78" modify_external_catalogs: true
I20240503 02:12:39.948544 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 41531ddcbf66465094d8f06e1116b3be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.950217 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:39.950708 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 95c7e4a2954e4161b9acc029873f923b
W20240503 02:12:39.950755 4746 leader_election.cc:341] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.950883 4746 leader_election.cc:302] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:39.951288 6296 raft_consensus.cc:2726] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.951447 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9d3caa7babae4afbaa0e5351cf84e421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.952021 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d3caa7babae4afbaa0e5351cf84e421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.952055 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c7e4a2954e4161b9acc029873f923b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:39.952636 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c7e4a2954e4161b9acc029873f923b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:39.953759 4494 tablet_service.cc:1467] Processing CreateTablet for tablet e2875ac004f5422fb1cb87d432694429 (DEFAULT_TABLE table=default.Table_b2cd99be75e54340bb07407e8831e25c [id=d066cd6738e34fb4a38a91e091e8873a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:39.954272 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91798b4bc2f74b9289425531937a78ce consensus state has no leader
I20240503 02:12:39.954659 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1aa10835245349bcade6a83360c1983d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:39.955243 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1aa10835245349bcade6a83360c1983d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:39.956102 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 17c2a8190210439fb7589cb02ec82bc2 (table default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae]). Expected version 4 got 0
I20240503 02:12:39.956377 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:39.957281 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c7e4a2954e4161b9acc029873f923b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.957625 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 41531ddcbf66465094d8f06e1116b3be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:39.957572 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 426cbf5ddc5a4944a487b305d5895eae (DEFAULT_TABLE table=default.Table_3fabf651e344476ca18d32ef633dd619 [id=c7e62260bd7c4dfc8b52df645339dcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.958132 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426cbf5ddc5a4944a487b305d5895eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.961841 4789 tablet_service.cc:1467] Processing CreateTablet for tablet e2875ac004f5422fb1cb87d432694429 (DEFAULT_TABLE table=default.Table_b2cd99be75e54340bb07407e8831e25c [id=d066cd6738e34fb4a38a91e091e8873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.962045 5115 ts_tablet_manager.cc:1888] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:39.962265 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2875ac004f5422fb1cb87d432694429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.962165 5115 log.cc:1196] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/4ced4dfb2cf541af98d50d22d1a0f0ac
I20240503 02:12:39.962764 5115 ts_tablet_manager.cc:1909] T 4ced4dfb2cf541af98d50d22d1a0f0ac P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:39.964380 5159 ts_tablet_manager.cc:1888] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:39.964468 5159 log.cc:1196] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3db94fb9d53044d796e2973143d888fb
I20240503 02:12:39.964718 5159 ts_tablet_manager.cc:1909] T 3db94fb9d53044d796e2973143d888fb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:39.964995 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 737) took 1321 ms. Trace:
I20240503 02:12:39.965068 5159 rpcz_store.cc:276] 0503 02:12:38.643183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.643442 (+ 259us) service_pool.cc:225] Handling call
0503 02:12:39.806306 (+1162864us) ts_tablet_manager.cc:1120] Deleting tablet 3db94fb9d53044d796e2973143d888fb
0503 02:12:39.806319 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.811990 (+ 5671us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.964294 (+152304us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.964976 (+ 682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162690}
I20240503 02:12:39.968612 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 959379aaf2d54b63bb265a6c8751b335 (DEFAULT_TABLE table=default.Table_8d03de02b1164ff5b180939da7d95ad3 [id=53d7ab88bb7e4f3ea5dae1c325eba741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.969395 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 959379aaf2d54b63bb265a6c8751b335 (DEFAULT_TABLE table=default.Table_8d03de02b1164ff5b180939da7d95ad3 [id=53d7ab88bb7e4f3ea5dae1c325eba741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.969766 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959379aaf2d54b63bb265a6c8751b335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.970247 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 663) took 2899 ms. Trace:
I20240503 02:12:39.970331 5115 rpcz_store.cc:276] 0503 02:12:37.070681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.074726 (+ 4045us) service_pool.cc:225] Handling call
0503 02:12:39.809084 (+2734358us) ts_tablet_manager.cc:1120] Deleting tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
0503 02:12:39.809093 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.927815 (+118722us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.961938 (+ 34123us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:39.970228 (+ 8290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2734191}
I20240503 02:12:39.973939 4495 tablet_service.cc:1467] Processing CreateTablet for tablet b5f444b22112407ca02fe86ac50167a9 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.974025 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet a8ce4b33f90c40c48cf0a538a1a7fde4 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:39.975049 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 959379aaf2d54b63bb265a6c8751b335 (DEFAULT_TABLE table=default.Table_8d03de02b1164ff5b180939da7d95ad3 [id=53d7ab88bb7e4f3ea5dae1c325eba741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.975140 4644 tablet_service.cc:1467] Processing CreateTablet for tablet e2875ac004f5422fb1cb87d432694429 (DEFAULT_TABLE table=default.Table_b2cd99be75e54340bb07407e8831e25c [id=d066cd6738e34fb4a38a91e091e8873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.976056 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959379aaf2d54b63bb265a6c8751b335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.976003 4785 tablet_service.cc:1467] Processing CreateTablet for tablet b5f444b22112407ca02fe86ac50167a9 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.976328 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f444b22112407ca02fe86ac50167a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.976586 5105 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:39.976701 5105 ts_tablet_manager.cc:1367] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.263s user 0.005s sys 0.006s
I20240503 02:12:39.977679 5105 raft_consensus.cc:348] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.979027 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 731) took 1338 ms. Trace:
I20240503 02:12:39.979099 5159 rpcz_store.cc:276] 0503 02:12:38.640600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.640649 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:39.977958 (+1337309us) ts_tablet_manager.cc:1120] Deleting tablet 5dbbfa2d1a49400c809d5b476036cdcd
0503 02:12:39.977967 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.979003 (+ 1036us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1332982}
I20240503 02:12:39.979434 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 732) took 1338 ms. Trace:
I20240503 02:12:39.979496 5159 rpcz_store.cc:276] 0503 02:12:38.641169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.641221 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:39.979347 (+1338126us) ts_tablet_manager.cc:1120] Deleting tablet 78b21a3010a4470288fbdd684ef64e77
0503 02:12:39.979357 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.979420 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334315}
I20240503 02:12:39.979755 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 734) took 1337 ms. Trace:
I20240503 02:12:39.979808 5159 rpcz_store.cc:276] 0503 02:12:38.641934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.641978 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:39.979693 (+1337715us) ts_tablet_manager.cc:1120] Deleting tablet ea4dac17dca3442d8934ac357779b78a
0503 02:12:39.979700 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:39.979745 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334626}
I20240503 02:12:39.979996 5159 tablet_replica.cc:330] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:39.980152 5159 raft_consensus.cc:2229] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.980280 5159 raft_consensus.cc:2258] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.980666 5105 raft_consensus.cc:374] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.980728 5105 raft_consensus.cc:729] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.980947 5105 consensus_queue.cc:260] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:39.981123 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2875ac004f5422fb1cb87d432694429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.984591 5105 ts_tablet_manager.cc:1398] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:39.984851 5105 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:39.985383 6262 raft_consensus.cc:482] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.985466 5159 ts_tablet_manager.cc:1875] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.985510 6262 raft_consensus.cc:504] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:39.986162 6262 leader_election.cc:288] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:39.991127 4608 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 7178760c21904207b6c8317093871f63
I20240503 02:12:39.995282 5115 tablet_replica.cc:330] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:39.995577 5115 raft_consensus.cc:2229] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:39.995805 5115 raft_consensus.cc:2258] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:39.996414 5115 ts_tablet_manager.cc:1875] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:39.999081 5105 tablet_bootstrap.cc:654] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.999962 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9801d79108b449c69664c004a6c352a9" } new_table_name: "default.Table_9988e5c156a84cc4a4fd9b745f10867a" modify_external_catalogs: true
I20240503 02:12:40.001173 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 41531ddcbf66465094d8f06e1116b3be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.001645 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cf90801bade4413b98880d6506b87076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.002154 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf90801bade4413b98880d6506b87076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.004956 4636 tablet_service.cc:1467] Processing CreateTablet for tablet b5f444b22112407ca02fe86ac50167a9 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.005265 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 810d37e454cb4633a099c546925eb48c (DEFAULT_TABLE table=default.Table_e474067bd616474aa34cb102a02b6b4d [id=8a6804808f9e401198db7de5361dfbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.005362 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f444b22112407ca02fe86ac50167a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.006654 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6 (DEFAULT_TABLE table=default.Table_2b3293557cf140d9a3edf888adb2f459 [id=abb0e49dc3044694b4bfb43f4a75d155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.008270 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.008886 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.009308 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.009363 5105 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.009467 5105 ts_tablet_manager.cc:1367] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 02:12:40.009718 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.010139 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.010263 5106 tablet_bootstrap.cc:654] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.010219 5105 raft_consensus.cc:348] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.010376 5105 raft_consensus.cc:374] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.010425 5105 raft_consensus.cc:729] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.010669 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.010618 5105 consensus_queue.cc:260] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.011114 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.011101 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 810d37e454cb4633a099c546925eb48c (DEFAULT_TABLE table=default.Table_e474067bd616474aa34cb102a02b6b4d [id=8a6804808f9e401198db7de5361dfbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.011457 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810d37e454cb4633a099c546925eb48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.011586 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.011839 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3db94fb9d53044d796e2973143d888fb (table default.Table_3e7dad4cf88048efa489e61c2595ad65 [id=5ca70228a70e4c02995e3c058a569e68]) successfully deleted
W20240503 02:12:40.012212 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5dbbfa2d1a49400c809d5b476036cdcd because the tablet was not found. No further retry: Not found: Tablet not found: 5dbbfa2d1a49400c809d5b476036cdcd
W20240503 02:12:40.012562 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 78b21a3010a4470288fbdd684ef64e77 because the tablet was not found. No further retry: Not found: Tablet not found: 78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:40.013012 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 810d37e454cb4633a099c546925eb48c (DEFAULT_TABLE table=default.Table_e474067bd616474aa34cb102a02b6b4d [id=8a6804808f9e401198db7de5361dfbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.013408 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810d37e454cb4633a099c546925eb48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.017154 5159 ts_tablet_manager.cc:1888] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:40.021476 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 59855ab3e42b4e019b769007ccbbc360 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:40.021963 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ea4dac17dca3442d8934ac357779b78a because the tablet was not found. No further retry: Not found: Tablet not found: ea4dac17dca3442d8934ac357779b78a
I20240503 02:12:40.023880 4493 tablet_service.cc:1467] Processing CreateTablet for tablet f7f35efaede44270a320f961119e2f17 (DEFAULT_TABLE table=default.Table_3e2c17a4387e47aa99c76caca5b18154 [id=1afea99e32d644aeb6ad7ce258816711]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.019896 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.025427 4497 tablet_service.cc:1467] Processing CreateTablet for tablet c5b10eef844a46b08a5d7e64eae8d51e (DEFAULT_TABLE table=default.Table_56a4b41e0c25434c814a607b0e6450b3 [id=aa84526e42c847db8d363da56c7abb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.025434 4636 tablet_service.cc:1467] Processing CreateTablet for tablet f7f35efaede44270a320f961119e2f17 (DEFAULT_TABLE table=default.Table_3e2c17a4387e47aa99c76caca5b18154 [id=1afea99e32d644aeb6ad7ce258816711]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.025790 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f35efaede44270a320f961119e2f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.021210 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6 (DEFAULT_TABLE table=default.Table_2b3293557cf140d9a3edf888adb2f459 [id=abb0e49dc3044694b4bfb43f4a75d155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.022058 5105 ts_tablet_manager.cc:1398] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:40.025694 5159 log.cc:1196] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/41d44fe8f6334572996665bdec60e2e5
I20240503 02:12:40.026731 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6 (DEFAULT_TABLE table=default.Table_2b3293557cf140d9a3edf888adb2f459 [id=abb0e49dc3044694b4bfb43f4a75d155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.027119 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.027513 5159 ts_tablet_manager.cc:1909] T 41d44fe8f6334572996665bdec60e2e5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:40.028218 5115 ts_tablet_manager.cc:1888] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.028539 5115 log.cc:1196] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/159ea1eb32394b04b72258d24196202b
I20240503 02:12:40.030464 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b10eef844a46b08a5d7e64eae8d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.030885 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810d37e454cb4633a099c546925eb48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.031133 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f444b22112407ca02fe86ac50167a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.031167 4608 consensus_peers.cc:485] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d022b5df4a914e439fff918f70c34157. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:40.031143 6262 raft_consensus.cc:482] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.031307 6262 raft_consensus.cc:504] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.034580 5115 ts_tablet_manager.cc:1909] T 159ea1eb32394b04b72258d24196202b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.034667 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 669) took 3079 ms. Trace:
I20240503 02:12:40.034754 5114 rpcz_store.cc:276] 0503 02:12:36.954793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.955054 (+ 261us) service_pool.cc:225] Handling call
0503 02:12:39.939400 (+2984346us) ts_tablet_manager.cc:1120] Deleting tablet 0a8c57ddf3d8428e8fa5ffb061dc054b
0503 02:12:39.939410 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.034648 (+ 95238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2984237}
I20240503 02:12:40.034912 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 671) took 2963 ms. Trace:
I20240503 02:12:40.034982 5115 rpcz_store.cc:276] 0503 02:12:37.074738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.074874 (+ 136us) service_pool.cc:225] Handling call
0503 02:12:39.970569 (+2895695us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:39.975799 (+ 5230us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.004076 (+ 28277us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.028118 (+ 24042us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.034893 (+ 6775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2895575}
I20240503 02:12:40.035282 5115 tablet_replica.cc:330] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.035470 5115 raft_consensus.cc:2229] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.035627 5115 raft_consensus.cc:2258] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.035825 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8b04b325acf94deea5af5e4a9660f7d5" } new_table_name: "default.Table_b145550ee4dc4b6a8f00b076f2ac72a5" modify_external_catalogs: true
I20240503 02:12:40.036175 6262 leader_election.cc:288] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:40.037076 4608 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:40.036000 4785 tablet_service.cc:1467] Processing CreateTablet for tablet c5b10eef844a46b08a5d7e64eae8d51e (DEFAULT_TABLE table=default.Table_56a4b41e0c25434c814a607b0e6450b3 [id=aa84526e42c847db8d363da56c7abb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.037364 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959379aaf2d54b63bb265a6c8751b335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.037719 5106 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.037834 5106 ts_tablet_manager.cc:1367] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.001s
I20240503 02:12:40.038159 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.038558 5115 ts_tablet_manager.cc:1875] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.038692 4644 tablet_service.cc:1467] Processing CreateTablet for tablet c5b10eef844a46b08a5d7e64eae8d51e (DEFAULT_TABLE table=default.Table_56a4b41e0c25434c814a607b0e6450b3 [id=aa84526e42c847db8d363da56c7abb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.039008 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b10eef844a46b08a5d7e64eae8d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.040987 5106 raft_consensus.cc:348] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.041464 5106 raft_consensus.cc:374] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.041639 5106 raft_consensus.cc:729] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.042062 5106 consensus_queue.cc:260] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.043272 5106 ts_tablet_manager.cc:1398] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:40.044584 5106 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.026445 5105 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.030045 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178760c21904207b6c8317093871f63" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.047423 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet 13e31219b2134fa080572238ff0be424 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:40.047637 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f35efaede44270a320f961119e2f17. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.048871 4608 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.049037 4608 leader_election.cc:302] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.025964 4789 tablet_service.cc:1467] Processing CreateTablet for tablet f7f35efaede44270a320f961119e2f17 (DEFAULT_TABLE table=default.Table_3e2c17a4387e47aa99c76caca5b18154 [id=1afea99e32d644aeb6ad7ce258816711]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.026326 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7bb94f4c2a45a0bd42e134fbccfde6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.049597 4608 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:40.049688 4608 leader_election.cc:302] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.050103 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2875ac004f5422fb1cb87d432694429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.050077 6374 raft_consensus.cc:2726] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.050226 6262 raft_consensus.cc:2726] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.051925 5105 tablet_bootstrap.cc:654] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.052933 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 735) took 1410 ms. Trace:
I20240503 02:12:40.053030 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b10eef844a46b08a5d7e64eae8d51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.053030 5159 rpcz_store.cc:276] 0503 02:12:38.642320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.642381 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:39.979957 (+1337576us) ts_tablet_manager.cc:1120] Deleting tablet 41d44fe8f6334572996665bdec60e2e5
0503 02:12:39.979964 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.003855 (+ 23891us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.017094 (+ 13239us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.052892 (+ 35798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334851}
I20240503 02:12:40.055020 5159 tablet_replica.cc:330] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:40.056154 5159 raft_consensus.cc:2229] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.056792 5159 raft_consensus.cc:2258] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.057049 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f35efaede44270a320f961119e2f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.057763 5159 ts_tablet_manager.cc:1875] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.066085 5106 tablet_bootstrap.cc:654] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.068320 5105 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.068670 5105 ts_tablet_manager.cc:1367] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240503 02:12:40.069387 5115 ts_tablet_manager.cc:1888] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.069555 5115 log.cc:1196] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/5f56701ed6424c60bf1a985eea1936fe
I20240503 02:12:40.069969 5115 ts_tablet_manager.cc:1909] T 5f56701ed6424c60bf1a985eea1936fe P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.070387 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 664) took 2999 ms. Trace:
I20240503 02:12:40.070462 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 666) took 3116 ms. Trace:
I20240503 02:12:40.070528 5115 rpcz_store.cc:276] 0503 02:12:37.070765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.074916 (+ 4151us) service_pool.cc:225] Handling call
0503 02:12:40.035239 (+2960323us) ts_tablet_manager.cc:1120] Deleting tablet 5f56701ed6424c60bf1a985eea1936fe
0503 02:12:40.035247 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.065320 (+ 30073us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.069289 (+ 3969us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.070364 (+ 1075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2960219}
I20240503 02:12:40.070596 5114 rpcz_store.cc:276] 0503 02:12:36.954455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:36.984327 (+ 29872us) service_pool.cc:225] Handling call
0503 02:12:40.035306 (+3050979us) ts_tablet_manager.cc:1120] Deleting tablet 7178760c21904207b6c8317093871f63
0503 02:12:40.035316 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.070425 (+ 35109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3050590}
I20240503 02:12:40.071182 5115 tablet_replica.cc:330] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.071437 5115 raft_consensus.cc:2229] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:40.072326 5115 raft_consensus.cc:2258] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.074036 5115 ts_tablet_manager.cc:1875] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.077853 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6ff638542714113aa75b6b69e0de916" } modify_external_catalogs: true
I20240503 02:12:40.077862 4778 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba9a44d78b4a649a6d358e5c187974 (DEFAULT_TABLE table=default.Table_2c9a62a49c0b48ef86699496645d06df [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.078091 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6ff638542714113aa75b6b69e0de916" } modify_external_catalogs: true
I20240503 02:12:40.078269 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba9a44d78b4a649a6d358e5c187974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.086231 4500 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba9a44d78b4a649a6d358e5c187974 (DEFAULT_TABLE table=default.Table_2c9a62a49c0b48ef86699496645d06df [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.095082 5105 raft_consensus.cc:348] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.098162 5103 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.098356 5106 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.098397 5103 ts_tablet_manager.cc:1367] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.256s user 0.021s sys 0.037s
I20240503 02:12:40.098682 5106 ts_tablet_manager.cc:1367] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.005s
I20240503 02:12:40.099196 5103 raft_consensus.cc:348] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.099499 5103 raft_consensus.cc:374] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.099637 5103 raft_consensus.cc:729] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.099951 5103 consensus_queue.cc:260] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.100193 5106 raft_consensus.cc:348] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.100575 5106 raft_consensus.cc:374] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.100620 5103 ts_tablet_manager.cc:1398] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:40.100747 5106 raft_consensus.cc:729] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.101517 5106 consensus_queue.cc:260] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.102739 5105 raft_consensus.cc:374] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.103340 5105 raft_consensus.cc:729] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.103715 5106 ts_tablet_manager.cc:1398] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:40.104233 5105 consensus_queue.cc:260] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.105144 5106 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.106477 5105 ts_tablet_manager.cc:1398] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240503 02:12:40.108150 5105 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.109812 5115 ts_tablet_manager.cc:1888] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.109984 5115 log.cc:1196] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/d022b5df4a914e439fff918f70c34157
I20240503 02:12:40.110527 5115 ts_tablet_manager.cc:1909] T d022b5df4a914e439fff918f70c34157 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.112161 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 665) took 3041 ms. Trace:
I20240503 02:12:40.112305 5115 rpcz_store.cc:276] 0503 02:12:37.070849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075041 (+ 4192us) service_pool.cc:225] Handling call
0503 02:12:40.071127 (+2996086us) ts_tablet_manager.cc:1120] Deleting tablet d022b5df4a914e439fff918f70c34157
0503 02:12:40.071136 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.100908 (+ 29772us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.109716 (+ 8808us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.112138 (+ 2422us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995964}
I20240503 02:12:40.112958 5115 tablet_replica.cc:330] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.113174 5115 raft_consensus.cc:2229] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:40.113572 5115 raft_consensus.cc:2258] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.114372 5106 tablet_bootstrap.cc:654] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.114586 5115 ts_tablet_manager.cc:1875] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.118081 5105 tablet_bootstrap.cc:654] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.111116 5159 ts_tablet_manager.cc:1888] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:40.122578 5159 log.cc:1196] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d39f042afb954daa923502ef12377bd5
I20240503 02:12:40.123826 5159 ts_tablet_manager.cc:1909] T d39f042afb954daa923502ef12377bd5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:40.126449 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 736) took 1483 ms. Trace:
I20240503 02:12:40.127456 6296 raft_consensus.cc:482] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.129033 6296 raft_consensus.cc:504] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.129698 6296 leader_election.cc:288] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.130162 4499 tablet_service.cc:1467] Processing CreateTablet for tablet bfe26248781b4173b4a0955629eb3857 (DEFAULT_TABLE table=default.Table_b94fa9b463444b4699a8781bcdca5f35 [id=1d370f7afda64def97053149ea330f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.131246 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:40.131474 4778 tablet_service.cc:1467] Processing CreateTablet for tablet bfe26248781b4173b4a0955629eb3857 (DEFAULT_TABLE table=default.Table_b94fa9b463444b4699a8781bcdca5f35 [id=1d370f7afda64def97053149ea330f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.131688 5115 ts_tablet_manager.cc:1888] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.131829 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe26248781b4173b4a0955629eb3857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.131850 5115 log.cc:1196] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/885129a4efc5475b97b7ac68e1db5e1d
W20240503 02:12:40.131973 4746 leader_election.cc:341] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.132210 5159 rpcz_store.cc:276] 0503 02:12:38.642726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.642782 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:40.054975 (+1412193us) ts_tablet_manager.cc:1120] Deleting tablet d39f042afb954daa923502ef12377bd5
0503 02:12:40.054986 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.066732 (+ 11746us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.111028 (+ 44296us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.126416 (+ 15388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409818}
I20240503 02:12:40.132524 5115 ts_tablet_manager.cc:1909] T 885129a4efc5475b97b7ac68e1db5e1d P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.133072 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 666) took 3062 ms. Trace:
I20240503 02:12:40.133210 5115 rpcz_store.cc:276] 0503 02:12:37.070936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075201 (+ 4265us) service_pool.cc:225] Handling call
0503 02:12:40.112837 (+3037636us) ts_tablet_manager.cc:1120] Deleting tablet 885129a4efc5475b97b7ac68e1db5e1d
0503 02:12:40.112846 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.121196 (+ 8350us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.131453 (+ 10257us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.133031 (+ 1578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3037485}
I20240503 02:12:40.133700 5115 tablet_replica.cc:330] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.134061 5115 raft_consensus.cc:2229] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:40.134591 5115 raft_consensus.cc:2258] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.135430 5115 ts_tablet_manager.cc:1875] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.135217 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 733) took 1493 ms. Trace:
I20240503 02:12:40.136117 5159 rpcz_store.cc:276] 0503 02:12:38.641709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.642794 (+ 1085us) service_pool.cc:225] Handling call
0503 02:12:40.134634 (+1491840us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:40.134643 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.135201 (+ 558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489462}
I20240503 02:12:40.137573 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 738) took 1492 ms. Trace:
I20240503 02:12:40.139266 5159 rpcz_store.cc:276] 0503 02:12:38.644864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.645413 (+ 549us) service_pool.cc:225] Handling call
0503 02:12:40.137490 (+1492077us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:40.137498 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.137560 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491903}
I20240503 02:12:40.140463 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 740) took 1439 ms. Trace:
I20240503 02:12:40.140767 5106 tablet_bootstrap.cc:492] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.140975 5106 ts_tablet_manager.cc:1367] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240503 02:12:40.141093 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.142351 5106 raft_consensus.cc:348] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.142671 5106 raft_consensus.cc:374] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.142747 5106 raft_consensus.cc:729] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.143218 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba9a44d78b4a649a6d358e5c187974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.143064 5106 consensus_queue.cc:260] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.143967 5106 ts_tablet_manager.cc:1398] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:40.144323 4644 tablet_service.cc:1467] Processing CreateTablet for tablet bfe26248781b4173b4a0955629eb3857 (DEFAULT_TABLE table=default.Table_b94fa9b463444b4699a8781bcdca5f35 [id=1d370f7afda64def97053149ea330f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.144701 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe26248781b4173b4a0955629eb3857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.145134 5106 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.145534 5103 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.146960 5103 tablet_bootstrap.cc:654] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.150825 5114 tablet_replica.cc:330] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:40.151072 5114 raft_consensus.cc:2229] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.151146 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe26248781b4173b4a0955629eb3857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.151352 5114 raft_consensus.cc:2258] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.153463 4644 tablet_service.cc:1467] Processing CreateTablet for tablet c294786084bc436f958f8e42c78ce3f9 (DEFAULT_TABLE table=default.Table_3f18eaa565c84fe18b01cefd0d501ad4 [id=3cdfe7dbe5c5442aae14d95a5d3c8b49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.153957 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c294786084bc436f958f8e42c78ce3f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.142783 4746 leader_election.cc:341] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.157238 4746 leader_election.cc:302] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.157485 6349 raft_consensus.cc:2726] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.145735 6296 raft_consensus.cc:482] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.157711 6296 raft_consensus.cc:504] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.158346 6296 leader_election.cc:288] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.159327 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.160061 4746 leader_election.cc:341] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.139184 5105 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.152804 4776 tablet_service.cc:1467] Processing CreateTablet for tablet c294786084bc436f958f8e42c78ce3f9 (DEFAULT_TABLE table=default.Table_3f18eaa565c84fe18b01cefd0d501ad4 [id=3cdfe7dbe5c5442aae14d95a5d3c8b49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.160394 5105 ts_tablet_manager.cc:1367] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240503 02:12:40.160693 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c294786084bc436f958f8e42c78ce3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.160789 5159 rpcz_store.cc:276] 0503 02:12:38.701532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.701591 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:40.140348 (+1438757us) ts_tablet_manager.cc:1120] Deleting tablet 17afd88bf6c040ad9e8d571eccd348dc
0503 02:12:40.140357 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.140447 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438537}
I20240503 02:12:40.161160 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 739) took 1460 ms. Trace:
I20240503 02:12:40.161229 5159 rpcz_store.cc:276] 0503 02:12:38.700850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.701792 (+ 942us) service_pool.cc:225] Handling call
0503 02:12:40.161056 (+1459264us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:40.161064 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.161143 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459035}
I20240503 02:12:40.161510 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 746) took 1017 ms. Trace:
I20240503 02:12:40.161567 5159 rpcz_store.cc:276] 0503 02:12:39.144210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.147407 (+ 3197us) service_pool.cc:225] Handling call
0503 02:12:40.161418 (+1014011us) ts_tablet_manager.cc:1120] Deleting tablet a4a16052d43d45aeb405c5de5dafa041
0503 02:12:40.161426 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.161498 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013827}
I20240503 02:12:40.161799 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 749) took 1005 ms. Trace:
I20240503 02:12:40.161851 5159 rpcz_store.cc:276] 0503 02:12:39.156307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.156365 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:40.161723 (+1005358us) ts_tablet_manager.cc:1120] Deleting tablet 16b3d1e7ce83425db0bd56457d9f2544
0503 02:12:40.161729 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.161788 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005085}
I20240503 02:12:40.162106 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 750) took 1001 ms. Trace:
I20240503 02:12:40.163417 6296 raft_consensus.cc:482] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.163522 6296 raft_consensus.cc:504] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.163455 5106 tablet_bootstrap.cc:654] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.164125 6296 leader_election.cc:288] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:40.165257 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.165480 4608 consensus_peers.cc:485] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7394c9d374404fb199ab6ff170d7febe. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:40.165791 4746 leader_election.cc:341] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.167425 5105 raft_consensus.cc:348] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.167482 6374 consensus_queue.cc:1035] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.168655 5105 raft_consensus.cc:374] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.169059 5105 raft_consensus.cc:729] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.169675 5105 consensus_queue.cc:260] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.171856 5105 ts_tablet_manager.cc:1398] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:40.173374 5105 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.175819 5105 tablet_bootstrap.cc:654] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.176481 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2 (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.176997 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.162173 5159 rpcz_store.cc:276] 0503 02:12:39.160594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.161051 (+ 457us) service_pool.cc:225] Handling call
0503 02:12:40.162030 (+1000979us) ts_tablet_manager.cc:1120] Deleting tablet a3374e095846463a859bb642c2f42167
0503 02:12:40.162036 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.162095 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000728}
I20240503 02:12:40.181830 5114 ts_tablet_manager.cc:1875] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.182353 5159 tablet_replica.cc:330] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:40.182587 5159 raft_consensus.cc:2229] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.182791 5159 raft_consensus.cc:2258] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.183521 5159 ts_tablet_manager.cc:1875] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.157840 4494 tablet_service.cc:1467] Processing CreateTablet for tablet c294786084bc436f958f8e42c78ce3f9 (DEFAULT_TABLE table=default.Table_3f18eaa565c84fe18b01cefd0d501ad4 [id=3cdfe7dbe5c5442aae14d95a5d3c8b49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.159898 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.161813 5103 tablet_bootstrap.cc:492] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.164785 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.184553 5103 ts_tablet_manager.cc:1367] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.005s
I20240503 02:12:40.176481 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2 (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.184906 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.184952 4746 leader_election.cc:341] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.185048 4746 leader_election.cc:302] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.185381 5103 raft_consensus.cc:348] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:40.185490 4746 leader_election.cc:341] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.185559 5103 raft_consensus.cc:374] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.185602 5103 raft_consensus.cc:729] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.185578 4746 leader_election.cc:302] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.185782 5103 consensus_queue.cc:260] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.185889 6349 raft_consensus.cc:2726] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.186040 6349 raft_consensus.cc:2726] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.186226 5103 ts_tablet_manager.cc:1398] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:40.186476 5103 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.186640 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c294786084bc436f958f8e42c78ce3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.187776 5103 tablet_bootstrap.cc:654] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.196462 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9348e65827449cad329aad4bc7a7ca (DEFAULT_TABLE table=default.Table_6b85445f36e344a8863969b82f94e867 [id=d4d03a12a5ea40eeb5d6b5a3937203db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.196521 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 372c128e62d04b7aa1bc3c07291d9876 (DEFAULT_TABLE table=default.Table_aab4f07f807f43f291c6bc552189e280 [id=8c3dedd890824a44838b7ace6f4388f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.196943 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372c128e62d04b7aa1bc3c07291d9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.197424 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9348e65827449cad329aad4bc7a7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.201110 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 372c128e62d04b7aa1bc3c07291d9876 (DEFAULT_TABLE table=default.Table_aab4f07f807f43f291c6bc552189e280 [id=8c3dedd890824a44838b7ace6f4388f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.201524 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372c128e62d04b7aa1bc3c07291d9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.202875 5115 ts_tablet_manager.cc:1888] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.204224 5115 log.cc:1196] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:40.208748 5115 ts_tablet_manager.cc:1909] T 22776c81d1a24d368abb68274cf32a0b P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.210880 4499 tablet_service.cc:1467] Processing CreateTablet for tablet b0e7985363654a298c91a0fe701cc3f7 (DEFAULT_TABLE table=default.Table_86543c68f82543d1bce59d92c6138b7b [id=56159a303fd14e05a91f327e7f46f2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.211335 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e7985363654a298c91a0fe701cc3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.211282 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 95c7e4a2954e4161b9acc029873f923b (DEFAULT_TABLE table=default.Table_9ffe0b87ba144949b561f827d3d06611 [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.211833 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c7e4a2954e4161b9acc029873f923b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.212793 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 4ced4dfb2cf541af98d50d22d1a0f0ac (table default.Table_25ca6a7de9b2413ba4623e94f133fac8 [id=65b0a82d0fbd4aa38786c55bb6aa3ee0]) successfully deleted
I20240503 02:12:40.213570 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 159ea1eb32394b04b72258d24196202b (table default.Table_e45d748d1bce468aac5ffaa88cf90f15 [id=8e56dd80d1434f44b3960fa13703b2b0]) successfully deleted
I20240503 02:12:40.213946 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 5f56701ed6424c60bf1a985eea1936fe (table default.Table_2fd053276a0848d6bdf3010494bcaa0b [id=1c4405572af54ee7b0f059b20fe7ec89]) successfully deleted
I20240503 02:12:40.214417 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet d022b5df4a914e439fff918f70c34157 (table default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]) successfully deleted
I20240503 02:12:40.214717 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 885129a4efc5475b97b7ac68e1db5e1d (table default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]) successfully deleted
I20240503 02:12:40.214890 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 95c7e4a2954e4161b9acc029873f923b (DEFAULT_TABLE table=default.Table_9ffe0b87ba144949b561f827d3d06611 [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.215291 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c7e4a2954e4161b9acc029873f923b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.215585 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a8c57ddf3d8428e8fa5ffb061dc054b already in progress: creating tablet
W20240503 02:12:40.215799 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7178760c21904207b6c8317093871f63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7178760c21904207b6c8317093871f63 already in progress: creating tablet
I20240503 02:12:40.216475 4775 tablet_service.cc:1467] Processing CreateTablet for tablet b0e7985363654a298c91a0fe701cc3f7 (DEFAULT_TABLE table=default.Table_86543c68f82543d1bce59d92c6138b7b [id=56159a303fd14e05a91f327e7f46f2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.216810 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e7985363654a298c91a0fe701cc3f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.218137 4608 consensus_peers.cc:485] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2152f205cf1b4ec7ae03605bbfa007e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:40.219466 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 667) took 3148 ms. Trace:
I20240503 02:12:40.210880 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 95c7e4a2954e4161b9acc029873f923b (DEFAULT_TABLE table=default.Table_9ffe0b87ba144949b561f827d3d06611 [id=5a08204f484f414c9749befba8b8182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.220834 5115 rpcz_store.cc:276] 0503 02:12:37.071004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075380 (+ 4376us) service_pool.cc:225] Handling call
0503 02:12:40.133626 (+3058246us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:40.133635 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.161713 (+ 28078us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.202160 (+ 40447us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.219142 (+ 16982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3058096}
I20240503 02:12:40.223405 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c7e4a2954e4161b9acc029873f923b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.225566 5115 tablet_replica.cc:330] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.228277 6374 raft_consensus.cc:482] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.228924 6374 raft_consensus.cc:504] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.228153 5115 raft_consensus.cc:2229] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.229220 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 372c128e62d04b7aa1bc3c07291d9876 (DEFAULT_TABLE table=default.Table_aab4f07f807f43f291c6bc552189e280 [id=8c3dedd890824a44838b7ace6f4388f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.229579 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372c128e62d04b7aa1bc3c07291d9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.229553 6374 leader_election.cc:288] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.230122 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.230326 4521 raft_consensus.cc:2454] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.230331 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:40.230526 4808 raft_consensus.cc:2454] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.230135 5115 raft_consensus.cc:2258] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.230731 4608 leader_election.cc:302] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.230782 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9348e65827449cad329aad4bc7a7ca (DEFAULT_TABLE table=default.Table_6b85445f36e344a8863969b82f94e867 [id=d4d03a12a5ea40eeb5d6b5a3937203db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.231119 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9348e65827449cad329aad4bc7a7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.231218 6374 raft_consensus.cc:2781] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.231285 6374 raft_consensus.cc:482] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.231343 6374 raft_consensus.cc:3037] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.233978 6374 raft_consensus.cc:504] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.234932 6374 leader_election.cc:288] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.235242 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:40.235420 4808 raft_consensus.cc:3037] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.235374 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59855ab3e42b4e019b769007ccbbc360" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:40.235531 4521 raft_consensus.cc:3037] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.236665 5115 ts_tablet_manager.cc:1875] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.237987 4521 raft_consensus.cc:2454] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:40.238058 4808 raft_consensus.cc:2454] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:40.238437 4608 leader_election.cc:302] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.238709 6374 raft_consensus.cc:2781] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.238974 6374 raft_consensus.cc:686] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:40.239321 6374 consensus_queue.cc:237] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.239488 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9348e65827449cad329aad4bc7a7ca (DEFAULT_TABLE table=default.Table_6b85445f36e344a8863969b82f94e867 [id=d4d03a12a5ea40eeb5d6b5a3937203db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.239893 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9348e65827449cad329aad4bc7a7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.241333 4636 tablet_service.cc:1467] Processing CreateTablet for tablet b0e7985363654a298c91a0fe701cc3f7 (DEFAULT_TABLE table=default.Table_86543c68f82543d1bce59d92c6138b7b [id=56159a303fd14e05a91f327e7f46f2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.243700 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e7985363654a298c91a0fe701cc3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.241658 4642 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba9a44d78b4a649a6d358e5c187974 (DEFAULT_TABLE table=default.Table_2c9a62a49c0b48ef86699496645d06df [id=5283ee29bf5142f0928e9aa919fb924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.246112 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba9a44d78b4a649a6d358e5c187974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.246376 6516 tablet.cc:1722] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:40.247133 5106 tablet_bootstrap.cc:492] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.247340 5106 ts_tablet_manager.cc:1367] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.008s
I20240503 02:12:40.248330 6515 tablet.cc:1722] T 2c5d58f97b58481ea1dfa75397a32ae2 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:40.250681 6349 raft_consensus.cc:482] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.250808 6349 raft_consensus.cc:504] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.251349 6349 leader_election.cc:288] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.257438 6349 raft_consensus.cc:482] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.257568 6349 raft_consensus.cc:504] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.257990 6513 tablet.cc:1722] T 2c5d58f97b58481ea1dfa75397a32ae2 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:40.258177 6349 leader_election.cc:288] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.258742 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:40.248330 5106 raft_consensus.cc:348] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.259287 5106 raft_consensus.cc:374] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.259392 5106 raft_consensus.cc:729] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.259510 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.259945 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035216c9475d4b59aa1ac81b285b4d8a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.260284 5106 consensus_queue.cc:260] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.261065 5106 ts_tablet_manager.cc:1398] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
W20240503 02:12:40.261224 4746 leader_election.cc:341] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:40.261416 4746 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.261441 5106 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:40.261546 4746 leader_election.cc:341] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.261615 4746 leader_election.cc:302] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.263252 5106 tablet_bootstrap.cc:654] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.263926 6349 raft_consensus.cc:2726] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:40.266135 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 50148.2us
I20240503 02:12:40.274935 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 41d44fe8f6334572996665bdec60e2e5 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:40.279985 6516 tablet.cc:1722] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:40.280035 5115 ts_tablet_manager.cc:1888] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.280964 5115 log.cc:1196] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/2f053537029a44d5a519b612d09d19a1
I20240503 02:12:40.283294 5115 ts_tablet_manager.cc:1909] T 2f053537029a44d5a519b612d09d19a1 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.284348 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035216c9475d4b59aa1ac81b285b4d8a" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.284996 4746 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.285105 4746 leader_election.cc:302] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.285290 6349 raft_consensus.cc:482] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.285429 6349 raft_consensus.cc:504] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.285390 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 668) took 3214 ms. Trace:
I20240503 02:12:40.285676 6296 raft_consensus.cc:2726] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.286058 6349 leader_election.cc:288] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.285835 5115 rpcz_store.cc:276] 0503 02:12:37.071076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075555 (+ 4479us) service_pool.cc:225] Handling call
0503 02:12:40.224868 (+3149313us) ts_tablet_manager.cc:1120] Deleting tablet 2f053537029a44d5a519b612d09d19a1
0503 02:12:40.224882 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.259062 (+ 34180us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.279480 (+ 20418us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.285211 (+ 5731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148846}
I20240503 02:12:40.286778 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38d63d2cc8c4f54ba43cdf8cde1841f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:40.287305 4746 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.288394 5115 tablet_replica.cc:330] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.289722 5115 raft_consensus.cc:2229] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.290973 5115 raft_consensus.cc:2258] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.294447 5115 ts_tablet_manager.cc:1875] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.297883 5114 ts_tablet_manager.cc:1888] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.298071 5114 log.cc:1196] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b161628065704c798020edbe098844c2
I20240503 02:12:40.298535 5114 ts_tablet_manager.cc:1909] T b161628065704c798020edbe098844c2 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:40.301484 4608 consensus_peers.cc:485] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a74f4816cad04a9eb5934aaba86c2fbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:40.304313 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet d39f042afb954daa923502ef12377bd5 (table default.Table_2c388fbe8fd148dc93f0d292381c0363 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]) successfully deleted
W20240503 02:12:40.304780 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because the tablet was not found. No further retry: Not found: Tablet not found: 22776c81d1a24d368abb68274cf32a0b
W20240503 02:12:40.305157 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 396b48865e954d05b9ba8084a2e7f29b because the tablet was not found. No further retry: Not found: Tablet not found: 396b48865e954d05b9ba8084a2e7f29b
W20240503 02:12:40.305505 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 17afd88bf6c040ad9e8d571eccd348dc because the tablet was not found. No further retry: Not found: Tablet not found: 17afd88bf6c040ad9e8d571eccd348dc
W20240503 02:12:40.305781 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 65774b69a8bd4664a6b55b84eba37f7f because the tablet was not found. No further retry: Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
W20240503 02:12:40.306041 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a4a16052d43d45aeb405c5de5dafa041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4a16052d43d45aeb405c5de5dafa041 already in progress: creating tablet
W20240503 02:12:40.306943 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a3374e095846463a859bb642c2f42167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3374e095846463a859bb642c2f42167 already in progress: creating tablet
I20240503 02:12:40.307685 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 22776c81d1a24d368abb68274cf32a0b (table default.Table_98ecda629a0e48c3bcb69d91efbfe2f4 [id=07ea56ce8d3749dfbcbc1a79ea84693c]) successfully deleted
I20240503 02:12:40.311734 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38d63d2cc8c4f54ba43cdf8cde1841f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.312532 4746 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.312716 4746 leader_election.cc:302] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.313105 6349 raft_consensus.cc:2726] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.322170 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 673) took 3178 ms. Trace:
I20240503 02:12:40.322355 5114 rpcz_store.cc:276] 0503 02:12:37.143333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.143558 (+ 225us) service_pool.cc:225] Handling call
0503 02:12:40.071121 (+2927563us) ts_tablet_manager.cc:1120] Deleting tablet b161628065704c798020edbe098844c2
0503 02:12:40.140966 (+ 69845us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.235909 (+ 94943us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.297769 (+ 61860us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.322146 (+ 24377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2649988}
I20240503 02:12:40.322990 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2 (DEFAULT_TABLE table=default.Table_41d45c372f894144a4e54926d63a14b8 [id=ba89a273edf145aa995821e3e06def8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.325206 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.325994 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 2f053537029a44d5a519b612d09d19a1 (table default.Table_26db706b59c346f59cb5df08c11c1d22 [id=d9011705ac294bd5832490cfdb4582b9]) successfully deleted
I20240503 02:12:40.326546 5114 tablet_replica.cc:330] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:40.326927 5114 raft_consensus.cc:2229] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.327122 5114 raft_consensus.cc:2258] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.330925 5105 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.331737 5105 ts_tablet_manager.cc:1367] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.001s
I20240503 02:12:40.332237 5159 ts_tablet_manager.cc:1888] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:40.337385 5114 ts_tablet_manager.cc:1875] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.337734 5159 log.cc:1196] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:40.339056 5159 ts_tablet_manager.cc:1909] T 81d6d3ca80c54c629a5de82b06a7aac4 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:40.339910 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 747) took 1183 ms. Trace:
W20240503 02:12:40.341642 4608 consensus_peers.cc:485] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f146f7cce0b24d3087bfe0973e2b7ef0. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:40.342816 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a11c306232c4e49ab02be0f195df935 consensus state has no leader
I20240503 02:12:40.343942 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b161628065704c798020edbe098844c2 (table default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]) successfully deleted
I20240503 02:12:40.332722 5105 raft_consensus.cc:348] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.344017 5159 rpcz_store.cc:276] 0503 02:12:39.156056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.172576 (+ 16520us) service_pool.cc:225] Handling call
0503 02:12:40.182300 (+1009724us) ts_tablet_manager.cc:1120] Deleting tablet 81d6d3ca80c54c629a5de82b06a7aac4
0503 02:12:40.182312 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.317431 (+135119us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.332128 (+ 14697us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.339890 (+ 7762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009521}
I20240503 02:12:40.344281 5105 raft_consensus.cc:374] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.344437 5105 raft_consensus.cc:729] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.344705 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 751) took 1129 ms. Trace:
I20240503 02:12:40.344705 5105 consensus_queue.cc:260] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.344857 5159 rpcz_store.cc:276] 0503 02:12:39.215681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.215738 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:40.344557 (+1128819us) ts_tablet_manager.cc:1120] Deleting tablet fc67bba49712445db6d784b6ed94dd7e
0503 02:12:40.344569 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.344692 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128644}
I20240503 02:12:40.345440 5105 ts_tablet_manager.cc:1398] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:12:40.348791 5105 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.350525 5105 tablet_bootstrap.cc:654] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.351621 5103 tablet_bootstrap.cc:492] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.352197 5103 ts_tablet_manager.cc:1367] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.016s
I20240503 02:12:40.354707 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_de40cf71b0974c799cf87de718e159b4" } modify_external_catalogs: true
I20240503 02:12:40.354919 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_de40cf71b0974c799cf87de718e159b4" } modify_external_catalogs: true
I20240503 02:12:40.352439 5159 tablet_replica.cc:330] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:40.356056 5159 raft_consensus.cc:2229] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:40.356379 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 289300b955dd41c09d2fc348224bb9f4 consensus state has no leader
I20240503 02:12:40.356781 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_949c4a7045e749d1839f5c9e29997ebd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:40.357630 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 13e31219b2134fa080572238ff0be424
I20240503 02:12:40.358176 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 73198ec769a9486da3f85e8fcbef8e78
W20240503 02:12:40.353281 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7099b64f41e54cd7b7967c52892b2200 consensus state has no leader
I20240503 02:12:40.361022 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 41ef3c6e5e7d40249b112144975ba220 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:40.361307 5159 raft_consensus.cc:2258] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.361690 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f6d3358d9b15463ca0f4ac45092a4616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:40.362349 5103 raft_consensus.cc:348] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.353823 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7" } modify_external_catalogs: true
I20240503 02:12:40.362987 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a2e82bdd3d844d90ab72fc4e5a3f7cb7" } modify_external_catalogs: true
I20240503 02:12:40.363803 5159 ts_tablet_manager.cc:1875] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.363940 5103 raft_consensus.cc:374] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:40.354218 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 478ac0f515b34e829c9a776bc32d875f consensus state has no leader
I20240503 02:12:40.364195 5103 raft_consensus.cc:729] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.364452 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9714c86da62e48489868fed877679b05" } modify_external_catalogs: true
I20240503 02:12:40.364637 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9714c86da62e48489868fed877679b05" } modify_external_catalogs: true
I20240503 02:12:40.366776 5106 tablet_bootstrap.cc:492] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.366979 5106 ts_tablet_manager.cc:1367] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.000s
I20240503 02:12:40.368098 5106 raft_consensus.cc:348] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.368358 5106 raft_consensus.cc:374] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.368475 5106 raft_consensus.cc:729] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.368752 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bfb8401dc518472abd4e26a2c788377d" } modify_external_catalogs: true
I20240503 02:12:40.354928 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 81d6d3ca80c54c629a5de82b06a7aac4 (table default.Table_27e6472c5d7746f59ee6399fa6db4d3b [id=74533d0ffe7340578b7b2201d27f7411]) successfully deleted
I20240503 02:12:40.368765 5106 consensus_queue.cc:260] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.369223 4268 catalog_manager.cc:6554] ReplaceTablet: tablet 13e31219b2134fa080572238ff0be424 deleted and replaced by tablet 26c505fdcb70418c85bb913a3be1d334
W20240503 02:12:40.369407 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fc67bba49712445db6d784b6ed94dd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc67bba49712445db6d784b6ed94dd7e already in progress: creating tablet
I20240503 02:12:40.369567 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a8465a99526148a0b0b88f92c5d1e3a0" } new_table_name: "default.Table_92682024602d4fdbae60fe91dc9b2651" modify_external_catalogs: true
I20240503 02:12:40.370306 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e31219b2134fa080572238ff0be424 with delete_type TABLET_DATA_DELETED (replaced by tablet 26c505fdcb70418c85bb913a3be1d334) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.371099 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:40.371237 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_949c4a7045e749d1839f5c9e29997ebd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.371282 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e31219b2134fa080572238ff0be424 with delete_type TABLET_DATA_DELETED (replaced by tablet 26c505fdcb70418c85bb913a3be1d334) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.364876 5103 consensus_queue.cc:260] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.371543 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.368949 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_bfb8401dc518472abd4e26a2c788377d" } modify_external_catalogs: true
I20240503 02:12:40.372702 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 73198ec769a9486da3f85e8fcbef8e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.373243 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e31219b2134fa080572238ff0be424 with delete_type TABLET_DATA_DELETED (replaced by tablet 26c505fdcb70418c85bb913a3be1d334) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.374351 5103 ts_tablet_manager.cc:1398] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
W20240503 02:12:40.376026 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6d3358d9b15463ca0f4ac45092a4616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.378271 5103 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.380115 5103 tablet_bootstrap.cc:654] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.387734 5105 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.387996 5105 ts_tablet_manager.cc:1367] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 02:12:40.388775 5105 raft_consensus.cc:348] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.389094 5105 raft_consensus.cc:374] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.389199 5105 raft_consensus.cc:729] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.389508 5105 consensus_queue.cc:260] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.390247 5105 ts_tablet_manager.cc:1398] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:40.390722 5105 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.392997 5105 tablet_bootstrap.cc:654] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.398145 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a8ecddfcfed2436e98bb7b6b38beb03f
I20240503 02:12:40.399080 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.399154 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_84278e0d870b4961a9ca3574619e6789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:40.399434 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.399646 5159 ts_tablet_manager.cc:1888] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:40.399780 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84278e0d870b4961a9ca3574619e6789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.399909 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ecddfcfed2436e98bb7b6b38beb03f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.399829 5159 log.cc:1196] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/4225c1bc133d4e18a4a47fd58820ca10
I20240503 02:12:40.400494 5159 ts_tablet_manager.cc:1909] T 4225c1bc133d4e18a4a47fd58820ca10 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:40.401886 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb2c0539bc1c4140b09ac51ad670c0e5 consensus state has no leader
I20240503 02:12:40.402431 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_aebba9d5877c44288da2f44d9e3b2e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.403053 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aebba9d5877c44288da2f44d9e3b2e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.403837 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3b92999890ee49d78d7c16f5ed1c4958
I20240503 02:12:40.404676 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3c1ab72c974d423a9a3a53f7cf411fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.405298 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c1ab72c974d423a9a3a53f7cf411fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.405344 6515 tablet.cc:1722] T 17c2a8190210439fb7589cb02ec82bc2 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240503 02:12:40.406386 4462 consensus_peers.cc:485] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7afb429d64497abcd9f1d0910651c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:40.407464 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cfbd41260f214e91ab9c1fec294cef84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.408022 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfbd41260f214e91ab9c1fec294cef84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.409289 6262 raft_consensus.cc:482] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.409420 6262 raft_consensus.cc:504] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.409845 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet fca2c34e0fc44be8b7ee9252ef557497
I20240503 02:12:40.410002 6262 leader_election.cc:288] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.410204 5106 ts_tablet_manager.cc:1398] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.043s user 0.000s sys 0.002s
I20240503 02:12:40.410463 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet c919bda1eaaa4e52828aac3ef337f3e5 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:40.412022 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:40.412245 4805 raft_consensus.cc:2454] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.412567 5106 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.413012 4608 leader_election.cc:302] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:40.413275 6262 raft_consensus.cc:482] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.413408 6262 raft_consensus.cc:504] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.414003 6262 leader_election.cc:288] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.414122 6262 raft_consensus.cc:2781] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.414171 6262 raft_consensus.cc:482] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.414230 6262 raft_consensus.cc:3037] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.414297 5106 tablet_bootstrap.cc:654] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.415751 4274 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_2df5b1aa30a0494f86117be55903a129 [id=ae1d450680154e00bb20d21e106dbd53] alter complete (version 2)
I20240503 02:12:40.416940 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.417161 4521 raft_consensus.cc:2454] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.417471 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.418110 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95681e67db1e4485876d530133e8a514" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:40.418393 4805 raft_consensus.cc:2379] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:40.419648 5114 ts_tablet_manager.cc:1888] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.419780 4608 leader_election.cc:302] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.419806 6262 raft_consensus.cc:504] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.419860 5114 log.cc:1196] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/885129a4efc5475b97b7ac68e1db5e1d
I20240503 02:12:40.420478 5114 ts_tablet_manager.cc:1909] T 885129a4efc5475b97b7ac68e1db5e1d P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:40.420475 6262 leader_election.cc:288] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.420629 6262 raft_consensus.cc:3037] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.423769 6262 raft_consensus.cc:2726] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.424765 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:40.425096 4813 raft_consensus.cc:3037] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.427579 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.428361 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.428776 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.425585 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28f66fe53064a46a0b98b78aa77115c" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:40.428589 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.428982 4525 raft_consensus.cc:3037] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.427762 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.430801 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.431979 5115 ts_tablet_manager.cc:1888] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.432045 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 675) took 3272 ms. Trace:
I20240503 02:12:40.433130 6516 tablet.cc:1722] T 17c2a8190210439fb7589cb02ec82bc2 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:40.435281 4525 raft_consensus.cc:2454] T b28f66fe53064a46a0b98b78aa77115c P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:40.435732 4608 leader_election.cc:302] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.435978 6317 raft_consensus.cc:2781] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.436116 6317 raft_consensus.cc:686] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:40.436414 6317 consensus_queue.cc:237] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.438021 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 07225ae9ad3c4e44ac0c5c51421b0115 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:40.441115 5115 log.cc:1196] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/36a14c4f785f4514ae74b8a0b3073503
I20240503 02:12:40.447687 4813 raft_consensus.cc:2454] T b28f66fe53064a46a0b98b78aa77115c P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:40.448539 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c919bda1eaaa4e52828aac3ef337f3e5
I20240503 02:12:40.449093 4272 catalog_manager.cc:6554] ReplaceTablet: tablet c919bda1eaaa4e52828aac3ef337f3e5 deleted and replaced by tablet ea11546176f94074a18b91d528290e72
I20240503 02:12:40.449539 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 with delete_type TABLET_DATA_DELETED (replaced by tablet ea11546176f94074a18b91d528290e72) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.449815 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 with delete_type TABLET_DATA_DELETED (replaced by tablet ea11546176f94074a18b91d528290e72) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.451104 5115 ts_tablet_manager.cc:1909] T 36a14c4f785f4514ae74b8a0b3073503 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.451762 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet c919bda1eaaa4e52828aac3ef337f3e5 with delete_type TABLET_DATA_DELETED (replaced by tablet ea11546176f94074a18b91d528290e72) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.453918 5106 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.454416 5106 ts_tablet_manager.cc:1367] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20240503 02:12:40.432819 5114 rpcz_store.cc:276] 0503 02:12:37.159360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.159420 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:40.322842 (+3163422us) ts_tablet_manager.cc:1120] Deleting tablet 885129a4efc5475b97b7ac68e1db5e1d
0503 02:12:40.326488 (+ 3646us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.340591 (+ 14103us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.419488 (+ 78897us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.432020 (+ 12532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2889742}
I20240503 02:12:40.459832 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 671) took 3317 ms. Trace:
I20240503 02:12:40.460047 5114 rpcz_store.cc:276] 0503 02:12:37.142525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.142583 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:40.456548 (+3313965us) ts_tablet_manager.cc:1120] Deleting tablet 19fb0929bd234788b5313d96d2e4845c
0503 02:12:40.459653 (+ 3105us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.459805 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950769}
I20240503 02:12:40.454126 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 669) took 3382 ms. Trace:
I20240503 02:12:40.460516 5115 rpcz_store.cc:276] 0503 02:12:37.071151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075739 (+ 4588us) service_pool.cc:225] Handling call
0503 02:12:40.288027 (+3212288us) ts_tablet_manager.cc:1120] Deleting tablet 36a14c4f785f4514ae74b8a0b3073503
0503 02:12:40.288036 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.334093 (+ 46057us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.431788 (+ 97695us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.453888 (+ 22100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212157}
I20240503 02:12:40.461133 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 689) took 2754 ms. Trace:
I20240503 02:12:40.461470 5115 tablet_replica.cc:330] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.461606 5114 rpcz_store.cc:276] 0503 02:12:37.706542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.706639 (+ 97us) service_pool.cc:225] Handling call
0503 02:12:40.460991 (+2754352us) ts_tablet_manager.cc:1120] Deleting tablet 6b15997a94b74ad3808d965acfb5d155
0503 02:12:40.461003 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.461113 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2754144}
I20240503 02:12:40.461799 5115 raft_consensus.cc:2229] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.462064 5115 raft_consensus.cc:2258] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.462373 5106 raft_consensus.cc:348] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.464305 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 690) took 2756 ms. Trace:
I20240503 02:12:40.464449 5114 rpcz_store.cc:276] 0503 02:12:37.708298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.708499 (+ 201us) service_pool.cc:225] Handling call
0503 02:12:40.462078 (+2753579us) ts_tablet_manager.cc:1120] Deleting tablet d84412b28b4c41c2a3ce18702403dcd8
0503 02:12:40.462089 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.464279 (+ 2190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753395}
I20240503 02:12:40.465351 5106 raft_consensus.cc:374] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.465507 5106 raft_consensus.cc:729] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.465673 6513 tablet.cc:1722] T 17c2a8190210439fb7589cb02ec82bc2 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 02:12:40.466035 5106 consensus_queue.cc:260] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.470589 6349 raft_consensus.cc:482] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.470767 6349 raft_consensus.cc:504] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.471449 6349 leader_election.cc:288] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.465804 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 691) took 2743 ms. Trace:
I20240503 02:12:40.472402 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.473130 5114 rpcz_store.cc:276] 0503 02:12:37.722853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.724257 (+ 1404us) service_pool.cc:225] Handling call
0503 02:12:40.465715 (+2741458us) ts_tablet_manager.cc:1120] Deleting tablet 8750b02c5936412db525d2d3e77befeb
0503 02:12:40.465725 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.465787 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2738373}
I20240503 02:12:40.465406 5115 ts_tablet_manager.cc:1875] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.471926 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.474562 4746 leader_election.cc:341] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:40.475354 4746 leader_election.cc:341] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.475452 4746 leader_election.cc:302] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.477711 6296 raft_consensus.cc:2726] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.478940 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 36a14c4f785f4514ae74b8a0b3073503 (table default.Table_489fd4b2341e48ee904ccad1bdad6d98 [id=e21aaa486bf4432cb49bea22320b96c3]) successfully deleted
I20240503 02:12:40.479481 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 885129a4efc5475b97b7ac68e1db5e1d (table default.Table_fa9bd69dd5d541339b74c4aeaa7fa7ba [id=1037fa1ee6644f8e8e1ff782c81c0a43]) successfully deleted
W20240503 02:12:40.479796 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 19fb0929bd234788b5313d96d2e4845c because the tablet was not found. No further retry: Not found: Tablet not found: 19fb0929bd234788b5313d96d2e4845c
W20240503 02:12:40.480032 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 6b15997a94b74ad3808d965acfb5d155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b15997a94b74ad3808d965acfb5d155 already in progress: creating tablet
W20240503 02:12:40.480265 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d84412b28b4c41c2a3ce18702403dcd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d84412b28b4c41c2a3ce18702403dcd8 already in progress: creating tablet
I20240503 02:12:40.474972 6448 consensus_queue.cc:1035] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.481168 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 694) took 2757 ms. Trace:
I20240503 02:12:40.481251 5114 rpcz_store.cc:276] 0503 02:12:37.723929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.727388 (+ 3459us) service_pool.cc:225] Handling call
0503 02:12:40.481066 (+2753678us) ts_tablet_manager.cc:1120] Deleting tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
0503 02:12:40.481077 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.481149 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753540}
I20240503 02:12:40.482293 5103 tablet_bootstrap.cc:492] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.482471 5103 ts_tablet_manager.cc:1367] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.003s
I20240503 02:12:40.482580 5114 tablet_replica.cc:330] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:40.485421 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8750b02c5936412db525d2d3e77befeb because the tablet was not found. No further retry: Not found: Tablet not found: 8750b02c5936412db525d2d3e77befeb
W20240503 02:12:40.485891 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c because the tablet was not found. No further retry: Not found: Tablet not found: 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:40.489145 5114 raft_consensus.cc:2229] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:40.489635 5114 raft_consensus.cc:2258] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.489742 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 757) took 1099 ms. Trace:
I20240503 02:12:40.483158 5103 raft_consensus.cc:348] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.490082 5103 raft_consensus.cc:374] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.490164 5103 raft_consensus.cc:729] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.490391 5103 consensus_queue.cc:260] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.490957 5103 ts_tablet_manager.cc:1398] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20240503 02:12:40.491420 5114 ts_tablet_manager.cc:1875] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.493605 5103 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.497421 5159 rpcz_store.cc:276] 0503 02:12:39.390210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.390339 (+ 129us) service_pool.cc:225] Handling call
0503 02:12:40.352396 (+962057us) ts_tablet_manager.cc:1120] Deleting tablet 4225c1bc133d4e18a4a47fd58820ca10
0503 02:12:40.352407 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.367919 (+ 15512us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.399500 (+ 31581us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.489714 (+ 90214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961886}
I20240503 02:12:40.497918 5114 ts_tablet_manager.cc:1888] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.498044 5114 log.cc:1196] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3b72c20a4b5a4d2ea266f031733d1aab
I20240503 02:12:40.499111 5106 ts_tablet_manager.cc:1398] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240503 02:12:40.499670 5106 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.504148 5106 tablet_bootstrap.cc:654] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.509948 5115 ts_tablet_manager.cc:1888] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.510175 5115 log.cc:1196] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/5dbbfa2d1a49400c809d5b476036cdcd
I20240503 02:12:40.510705 5115 ts_tablet_manager.cc:1909] T 5dbbfa2d1a49400c809d5b476036cdcd P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.513445 5103 tablet_bootstrap.cc:654] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.514492 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 670) took 3443 ms. Trace:
I20240503 02:12:40.515419 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 4225c1bc133d4e18a4a47fd58820ca10 (table default.Table_bb63ba232a074c0692fa657db43e25dc [id=5cbfe61d82174908b3b00ff5d287eb8f]) successfully deleted
I20240503 02:12:40.515937 5115 rpcz_store.cc:276] 0503 02:12:37.071222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.075888 (+ 4666us) service_pool.cc:225] Handling call
0503 02:12:40.461348 (+3385460us) ts_tablet_manager.cc:1120] Deleting tablet 5dbbfa2d1a49400c809d5b476036cdcd
0503 02:12:40.461359 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.506218 (+ 44859us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.509843 (+ 3625us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.514467 (+ 4624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3385281}
I20240503 02:12:40.523506 5114 ts_tablet_manager.cc:1909] T 3b72c20a4b5a4d2ea266f031733d1aab P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:40.523743 5115 tablet_replica.cc:330] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.524014 5115 raft_consensus.cc:2229] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.524322 5115 raft_consensus.cc:2258] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.525086 5115 ts_tablet_manager.cc:1875] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.526086 5106 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.526229 5106 ts_tablet_manager.cc:1367] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20240503 02:12:40.526907 5106 raft_consensus.cc:348] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.527093 5106 raft_consensus.cc:374] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.527141 5106 raft_consensus.cc:729] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.527406 5105 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.527374 5106 consensus_queue.cc:260] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.528041 5106 ts_tablet_manager.cc:1398] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:40.528455 5106 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.530112 5106 tablet_bootstrap.cc:654] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.533093 5105 ts_tablet_manager.cc:1367] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.004s
I20240503 02:12:40.534116 5105 raft_consensus.cc:348] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.534454 5105 raft_consensus.cc:374] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.534688 5105 raft_consensus.cc:729] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.535245 5105 consensus_queue.cc:260] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.536332 5105 ts_tablet_manager.cc:1398] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:40.536549 5159 tablet_replica.cc:330] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:40.521322 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 5dbbfa2d1a49400c809d5b476036cdcd (table default.Table_de35f554f3f64611a5a03c54e5c17e04 [id=218c0b2dedd3489a8c3bf6a508a52912]) successfully deleted
I20240503 02:12:40.537320 5159 raft_consensus.cc:2229] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.537707 5159 raft_consensus.cc:2258] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.538340 5159 ts_tablet_manager.cc:1875] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.545099 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 693) took 2821 ms. Trace:
I20240503 02:12:40.545328 5114 rpcz_store.cc:276] 0503 02:12:37.723658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.727562 (+ 3904us) service_pool.cc:225] Handling call
0503 02:12:40.481495 (+2753933us) ts_tablet_manager.cc:1120] Deleting tablet 3b72c20a4b5a4d2ea266f031733d1aab
0503 02:12:40.482524 (+ 1029us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.494697 (+ 12173us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.497806 (+ 3109us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.545067 (+ 47261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753622}
I20240503 02:12:40.545965 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 692) took 2822 ms. Trace:
I20240503 02:12:40.546061 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 3b72c20a4b5a4d2ea266f031733d1aab (table default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]) successfully deleted
I20240503 02:12:40.546108 5114 rpcz_store.cc:276] 0503 02:12:37.723413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.727906 (+ 4493us) service_pool.cc:225] Handling call
0503 02:12:40.545852 (+2817946us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:40.545864 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.545952 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2817670}
I20240503 02:12:40.546969 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 699) took 2465 ms. Trace:
W20240503 02:12:40.547045 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 159ea1eb32394b04b72258d24196202b because the tablet was not found. No further retry: Not found: Tablet not found: 159ea1eb32394b04b72258d24196202b
I20240503 02:12:40.547109 5114 rpcz_store.cc:276] 0503 02:12:38.081452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.090561 (+ 9109us) service_pool.cc:225] Handling call
0503 02:12:40.546843 (+2456282us) ts_tablet_manager.cc:1120] Deleting tablet 8f8b6599206549dd8ddf449a52c61fce
0503 02:12:40.546852 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.546952 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456037}
I20240503 02:12:40.547569 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 698) took 2466 ms. Trace:
I20240503 02:12:40.547708 5114 rpcz_store.cc:276] 0503 02:12:38.081333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.091117 (+ 9784us) service_pool.cc:225] Handling call
0503 02:12:40.547457 (+2456340us) ts_tablet_manager.cc:1120] Deleting tablet 0e89c1cf6ed54d2fafb2173ea9373ab5
0503 02:12:40.547467 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.547555 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456104}
I20240503 02:12:40.548116 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 701) took 2284 ms. Trace:
I20240503 02:12:40.548251 5114 rpcz_store.cc:276] 0503 02:12:38.263423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.264290 (+ 867us) service_pool.cc:225] Handling call
0503 02:12:40.548021 (+2283731us) ts_tablet_manager.cc:1120] Deleting tablet 20b6a18be64049049e1a083bf4af51ba
0503 02:12:40.548029 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.548102 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283523}
I20240503 02:12:40.548667 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 700) took 2286 ms. Trace:
W20240503 02:12:40.547626 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8f8b6599206549dd8ddf449a52c61fce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8b6599206549dd8ddf449a52c61fce already in progress: creating tablet
W20240503 02:12:40.552394 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 already in progress: creating tablet
W20240503 02:12:40.552781 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 20b6a18be64049049e1a083bf4af51ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b6a18be64049049e1a083bf4af51ba already in progress: creating tablet
I20240503 02:12:40.557291 5105 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.559199 5105 tablet_bootstrap.cc:654] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.562861 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0d7ac590920b4d4fb4fbc65cf513ef74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.563524 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7ac590920b4d4fb4fbc65cf513ef74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.564735 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2df5b1aa30a0494f86117be55903a129" } new_table_name: "default.Table_1d13d28ae8bc4af08a1b447dc0e62c6b" modify_external_catalogs: true
I20240503 02:12:40.566002 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c5f5ee35b528431cb4e7256693a743a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.566480 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5f5ee35b528431cb4e7256693a743a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.567710 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cea7d864ad774bb5b4b975e8e72989ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.568226 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea7d864ad774bb5b4b975e8e72989ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.569639 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_af6784c632dc44038b2633d2a6ccd1cd" } new_table_name: "default.Table_bc90b6bcb7514fbcace5a7639189ae7c" modify_external_catalogs: true
I20240503 02:12:40.585477 5114 rpcz_store.cc:276] 0503 02:12:38.261751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.265244 (+ 3493us) service_pool.cc:225] Handling call
0503 02:12:40.548564 (+2283320us) ts_tablet_manager.cc:1120] Deleting tablet 56aa7303f7784d2289a8da9653d1e317
0503 02:12:40.548576 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.548651 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283101}
I20240503 02:12:40.586073 5114 tablet_replica.cc:330] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:40.586086 5106 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.586256 5106 ts_tablet_manager.cc:1367] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.002s
W20240503 02:12:40.586356 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 56aa7303f7784d2289a8da9653d1e317 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56aa7303f7784d2289a8da9653d1e317 already in progress: creating tablet
I20240503 02:12:40.586388 5114 raft_consensus.cc:2229] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.586591 5114 raft_consensus.cc:2258] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.587065 5106 raft_consensus.cc:348] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.587226 5114 ts_tablet_manager.cc:1875] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:40.587289 5106 raft_consensus.cc:374] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.587350 5106 raft_consensus.cc:729] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.587679 5106 consensus_queue.cc:260] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.588326 5106 ts_tablet_manager.cc:1398] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:40.588642 6349 raft_consensus.cc:482] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.588819 5106 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.588805 6349 raft_consensus.cc:504] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.589525 6349 leader_election.cc:288] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.590763 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.591517 5106 tablet_bootstrap.cc:654] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:40.591575 4746 leader_election.cc:341] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.592221 6478 consensus_queue.cc:1035] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.594332 5103 tablet_bootstrap.cc:492] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.594604 5103 ts_tablet_manager.cc:1367] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.009s
I20240503 02:12:40.595130 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.595775 4746 leader_election.cc:341] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.595909 4746 leader_election.cc:302] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.596448 6349 raft_consensus.cc:2726] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.596402 5103 raft_consensus.cc:348] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.596767 5103 raft_consensus.cc:374] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.596966 5103 raft_consensus.cc:729] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.597406 5103 consensus_queue.cc:260] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.600965 5115 ts_tablet_manager.cc:1888] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.600848 5103 ts_tablet_manager.cc:1398] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:40.601151 5115 log.cc:1196] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3b41dce88919456aa846dbdcf5e2759f
I20240503 02:12:40.601563 5115 ts_tablet_manager.cc:1909] T 3b41dce88919456aa846dbdcf5e2759f P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:40.601933 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 662) took 3531 ms. Trace:
I20240503 02:12:40.602042 5115 rpcz_store.cc:276] 0503 02:12:37.070496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.070539 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:40.521566 (+3451027us) ts_tablet_manager.cc:1120] Deleting tablet 3b41dce88919456aa846dbdcf5e2759f
0503 02:12:40.523673 (+ 2107us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.555381 (+ 31708us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.595135 (+ 39754us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.601913 (+ 6778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3440745}
I20240503 02:12:40.602981 5103 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.608562 5114 ts_tablet_manager.cc:1888] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:40.621587 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7abfa4c4ae61433f9ac98e07d9954442"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.622211 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7abfa4c4ae61433f9ac98e07d9954442 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.622625 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_53cb05a18acb413295d4cfe37fb3b76d" } new_table_name: "default.Table_c24f2a34ca8a43a388947bcc6a82f54e" modify_external_catalogs: true
I20240503 02:12:40.624220 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f07e8b862b79406bab7c9aa1e63fd2f8" } modify_external_catalogs: true
I20240503 02:12:40.624413 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f07e8b862b79406bab7c9aa1e63fd2f8" } modify_external_catalogs: true
W20240503 02:12:40.625823 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8425bb6a789a4aaf8c6e3714943852cc has no consensus state
I20240503 02:12:40.626226 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_670774410ede43baac53339c1d242797" } new_table_name: "default.Table_415ae7620c844cd1a4e2190eb2efe4c2" modify_external_catalogs: true
I20240503 02:12:40.626483 6349 raft_consensus.cc:482] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.626639 6349 raft_consensus.cc:504] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.627277 6349 leader_election.cc:288] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.631263 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_681b4fa6666c4e9aaddf611003dcaa44" } modify_external_catalogs: true
I20240503 02:12:40.631541 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_681b4fa6666c4e9aaddf611003dcaa44" } modify_external_catalogs: true
I20240503 02:12:40.632493 5105 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.632690 5105 ts_tablet_manager.cc:1367] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.002s
I20240503 02:12:40.633599 5105 raft_consensus.cc:348] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.633857 5105 raft_consensus.cc:374] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.633970 5105 raft_consensus.cc:729] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.634248 5105 consensus_queue.cc:260] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.635052 5105 ts_tablet_manager.cc:1398] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:40.637065 5105 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.615550 5106 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:40.637426 5106 ts_tablet_manager.cc:1367] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20240503 02:12:40.637545 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 3b41dce88919456aa846dbdcf5e2759f (table default.Table_02838524663542819d76fcab9d549504 [id=815f6c57977246f4a02f7bf6ca484c9e]) successfully deleted
I20240503 02:12:40.638180 5106 raft_consensus.cc:348] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.638376 5106 raft_consensus.cc:374] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.638430 5106 raft_consensus.cc:729] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.603402 5159 ts_tablet_manager.cc:1888] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:40.638700 5159 log.cc:1196] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/6c9090b11c8a4bf0b4b92523b9431b79
I20240503 02:12:40.638634 5106 consensus_queue.cc:260] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.639107 5106 ts_tablet_manager.cc:1398] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:40.628005 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.640652 4746 leader_election.cc:341] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.641523 5159 ts_tablet_manager.cc:1909] T 6c9090b11c8a4bf0b4b92523b9431b79 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:40.646673 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 758) took 1255 ms. Trace:
I20240503 02:12:40.654300 5159 rpcz_store.cc:276] 0503 02:12:39.391562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.391912 (+ 350us) service_pool.cc:225] Handling call
0503 02:12:40.516465 (+1124553us) ts_tablet_manager.cc:1120] Deleting tablet 6c9090b11c8a4bf0b4b92523b9431b79
0503 02:12:40.516475 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.582106 (+ 65631us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.603264 (+ 21158us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.646629 (+ 43365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124393}
I20240503 02:12:40.653340 6349 raft_consensus.cc:482] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.656365 6349 raft_consensus.cc:504] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.656577 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 6c9090b11c8a4bf0b4b92523b9431b79 (table default.Table_2c9a62a49c0b48ef86699496645d06df [id=5283ee29bf5142f0928e9aa919fb924a]) successfully deleted
I20240503 02:12:40.650138 5105 tablet_bootstrap.cc:654] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.657155 6349 leader_election.cc:288] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.658223 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 764) took 1187 ms. Trace:
I20240503 02:12:40.658298 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:40.658905 4746 leader_election.cc:341] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.658766 5159 rpcz_store.cc:276] 0503 02:12:39.470597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.481017 (+ 10420us) service_pool.cc:225] Handling call
0503 02:12:40.658056 (+1177039us) ts_tablet_manager.cc:1120] Deleting tablet 1716c4807e004760b34c0492cb7f522d
0503 02:12:40.658067 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.658183 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176806}
W20240503 02:12:40.659965 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1716c4807e004760b34c0492cb7f522d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1716c4807e004760b34c0492cb7f522d already in progress: creating tablet
I20240503 02:12:40.660456 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 765) took 1125 ms. Trace:
I20240503 02:12:40.662369 5159 rpcz_store.cc:276] 0503 02:12:39.534633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.534700 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:40.660313 (+1125613us) ts_tablet_manager.cc:1120] Deleting tablet 5e534255f170416690c18ca6bd571117
0503 02:12:40.660323 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.660413 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125335}
W20240503 02:12:40.668323 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5e534255f170416690c18ca6bd571117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e534255f170416690c18ca6bd571117 already in progress: creating tablet
I20240503 02:12:40.669751 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 766) took 1132 ms. Trace:
I20240503 02:12:40.669994 6516 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 02:12:40.680544 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 192460518d49417cb8f03ac6ec971eef consensus state has no leader
I20240503 02:12:40.680995 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c77b55b8023b459c96d97e8d9c8f7986" } new_table_name: "default.Table_d199ea94999f4b009ecca81b9c654f93" modify_external_catalogs: true
W20240503 02:12:40.682588 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9522b13ac2624c9f95b05ab335ae02da consensus state has no leader
I20240503 02:12:40.682945 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ef5ce757a4e349cab1cdea6c1bc22736" } modify_external_catalogs: true
I20240503 02:12:40.683126 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_ef5ce757a4e349cab1cdea6c1bc22736" } modify_external_catalogs: true
I20240503 02:12:40.684626 6515 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:40.684816 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_656a2eccad1e4e59aa586fc88c5d7f89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.685506 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_656a2eccad1e4e59aa586fc88c5d7f89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.685626 5105 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.686033 5105 ts_tablet_manager.cc:1367] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20240503 02:12:40.687031 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 07225ae9ad3c4e44ac0c5c51421b0115
I20240503 02:12:40.687660 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 07225ae9ad3c4e44ac0c5c51421b0115 deleted and replaced by tablet 2da27c48da204cb384a99b0032029b63
I20240503 02:12:40.687974 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_799bd817690849dc9654a7fb51482f2b" } modify_external_catalogs: true
I20240503 02:12:40.688153 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_799bd817690849dc9654a7fb51482f2b" } modify_external_catalogs: true
I20240503 02:12:40.688558 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da27c48da204cb384a99b0032029b63) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.685690 5159 rpcz_store.cc:276] 0503 02:12:39.537505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.537556 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:40.669592 (+1132036us) ts_tablet_manager.cc:1120] Deleting tablet f20e573d70bd46aab33466ccfc801465
0503 02:12:40.669604 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.669704 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131837}
W20240503 02:12:40.689404 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f20e573d70bd46aab33466ccfc801465 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20e573d70bd46aab33466ccfc801465 already in progress: creating tablet
I20240503 02:12:40.689468 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0ab2b01fb13c4aaa931b3a633ab5d847
I20240503 02:12:40.690233 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.690264 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_348455618e27462284c9005cc2368dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.690781 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_348455618e27462284c9005cc2368dd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.690987 5114 log.cc:1196] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/2152f205cf1b4ec7ae03605bbfa007e1
I20240503 02:12:40.690972 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.691342 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da27c48da204cb384a99b0032029b63) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.691893 5114 ts_tablet_manager.cc:1909] T 2152f205cf1b4ec7ae03605bbfa007e1 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:40.692112 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 676d5f52102d496081df3964722b286f
I20240503 02:12:40.692827 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8e5698a54a9249b2a78432ae0801f067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.693347 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e5698a54a9249b2a78432ae0801f067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.697829 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 676d5f52102d496081df3964722b286f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.698700 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e366ce9d79b3458fb26e96b6e9278fbe (table default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9]). Expected version 2 got 0
I20240503 02:12:40.698999 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 289300b955dd41c09d2fc348224bb9f4 (table default.Table_b145550ee4dc4b6a8f00b076f2ac72a5 [id=2817ff73927a4593bddef5408db5cc49]). Expected version 1 got 0
I20240503 02:12:40.699167 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:40.699332 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:40.699543 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:40.700062 4275 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_c7106f5dd350475781a5c67d7c4a6b30 [id=7dd3df315372418f9c30f7a0e506fdd9] alter complete (version 2)
I20240503 02:12:40.701543 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 22776c81d1a24d368abb68274cf32a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.701884 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 78e9faaa6a584ec38b9da9d2670e47f3
I20240503 02:12:40.705565 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.705813 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1a6867e7591e44dc942982feb523e922"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.706336 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a6867e7591e44dc942982feb523e922 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.707701 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4b3d4e5854b34468a48715e9093f70e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.708185 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b3d4e5854b34468a48715e9093f70e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.710295 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ecb8bc8e313042c292129fa02342ed2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.710785 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb8bc8e313042c292129fa02342ed2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.716430 6349 raft_consensus.cc:482] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.716630 6349 raft_consensus.cc:504] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:40.717443 4608 consensus_peers.cc:485] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:40.717875 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e49cde2fd4aab9f44921d3c3921a5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:40.717991 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 676d5f52102d496081df3964722b286f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.718235 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.718400 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:40.718482 4746 leader_election.cc:341] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.718546 6349 leader_election.cc:288] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.718660 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 772) took 1051 ms. Trace:
I20240503 02:12:40.718794 5159 rpcz_store.cc:276] 0503 02:12:39.640222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.640271 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:40.691155 (+1050884us) ts_tablet_manager.cc:1120] Deleting tablet 159ea1eb32394b04b72258d24196202b
0503 02:12:40.691167 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.691247 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050666}
W20240503 02:12:40.719364 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 159ea1eb32394b04b72258d24196202b because the tablet was not found. No further retry: Not found: Tablet not found: 159ea1eb32394b04b72258d24196202b
I20240503 02:12:40.720870 5105 raft_consensus.cc:348] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.721175 5105 raft_consensus.cc:374] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.721279 5105 raft_consensus.cc:729] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.721839 5105 consensus_queue.cc:260] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.722574 5105 ts_tablet_manager.cc:1398] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 02:12:40.659585 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.663646 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e (DEFAULT_TABLE table=default.Table_45a63336226e4bd29bf8fb7d9c8e02a2 [id=a074606d154b4677903fc7ce4c194cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.665724 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.669359 6513 tablet.cc:1722] T b23be1160c454c139128aff5a960d183 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 02:12:40.723363 4746 leader_election.cc:341] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.723484 4746 leader_election.cc:302] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.688892 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 07225ae9ad3c4e44ac0c5c51421b0115 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da27c48da204cb384a99b0032029b63) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.690709 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.613317 5103 tablet_bootstrap.cc:654] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.698330 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 676d5f52102d496081df3964722b286f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.701786 6478 consensus_queue.cc:1035] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.705367 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e9faaa6a584ec38b9da9d2670e47f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.717754 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e49cde2fd4aab9f44921d3c3921a5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.721603 6448 raft_consensus.cc:482] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.724558 6448 raft_consensus.cc:504] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.725186 6448 leader_election.cc:288] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:40.725354 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.726217 6349 raft_consensus.cc:2726] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.726554 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:40.726895 4675 raft_consensus.cc:2454] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:40.727249 4462 leader_election.cc:302] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.727538 6478 raft_consensus.cc:2781] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.727602 6478 raft_consensus.cc:482] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.727654 6478 raft_consensus.cc:3037] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.730237 6478 raft_consensus.cc:504] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.730995 6478 leader_election.cc:288] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:40.731781 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:40.731961 4746 leader_election.cc:341] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.731998 4667 raft_consensus.cc:3037] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.735193 4667 raft_consensus.cc:2454] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:40.735680 6374 raft_consensus.cc:482] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.735800 6374 raft_consensus.cc:504] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.736403 6374 leader_election.cc:288] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.737433 4462 leader_election.cc:302] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.737457 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94fb9d53044d796e2973143d888fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.737717 4525 raft_consensus.cc:2454] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.738173 4608 leader_election.cc:302] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.738517 6374 raft_consensus.cc:2781] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.738605 6374 raft_consensus.cc:482] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.738684 6374 raft_consensus.cc:3037] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.741071 6478 raft_consensus.cc:2781] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.741325 6478 raft_consensus.cc:686] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:40.741632 6478 consensus_queue.cc:237] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.742321 6374 raft_consensus.cc:504] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.742956 6374 leader_election.cc:288] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.743929 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:40.744216 4805 raft_consensus.cc:2454] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:40.744398 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e713ee37d042309166b0f118e14986" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:40.744587 4808 raft_consensus.cc:3037] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:40.744735 4608 leader_election.cc:341] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3db94fb9d53044d796e2973143d888fb
I20240503 02:12:40.747460 4808 raft_consensus.cc:2454] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:40.748508 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94fb9d53044d796e2973143d888fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:40.748701 4525 raft_consensus.cc:3037] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.751397 4525 raft_consensus.cc:2454] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:40.751943 4608 leader_election.cc:341] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3db94fb9d53044d796e2973143d888fb
I20240503 02:12:40.752763 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 773) took 1111 ms. Trace:
I20240503 02:12:40.754333 4608 leader_election.cc:302] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:40.755016 5159 rpcz_store.cc:276] 0503 02:12:39.640871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.640945 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:40.719710 (+1078765us) ts_tablet_manager.cc:1120] Deleting tablet b161628065704c798020edbe098844c2
0503 02:12:40.743714 (+ 24004us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.752728 (+ 9014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":1078612}
I20240503 02:12:40.755283 6317 raft_consensus.cc:2781] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.757107 4525 raft_consensus.cc:3037] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.759121 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e (DEFAULT_TABLE table=default.Table_45a63336226e4bd29bf8fb7d9c8e02a2 [id=a074606d154b4677903fc7ce4c194cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.759708 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 (DEFAULT_TABLE table=default.Table_817b839a30804e609f4916cf44d4dafd [id=95a00c60dd054e6fac7bcd616a3d82b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.760095 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4dabd6e05948b98f02515023e2aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.762941 6374 consensus_queue.cc:1035] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.758617 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e (DEFAULT_TABLE table=default.Table_45a63336226e4bd29bf8fb7d9c8e02a2 [id=a074606d154b4677903fc7ce4c194cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.763661 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.766786 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b23be1160c454c139128aff5a960d183 (table default.Table_1d13d28ae8bc4af08a1b447dc0e62c6b [id=ae1d450680154e00bb20d21e106dbd53]). Expected version 3 got 2
W20240503 02:12:40.772030 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b161628065704c798020edbe098844c2 because the tablet was not found. No further retry: Not found: Tablet not found: b161628065704c798020edbe098844c2
I20240503 02:12:40.772603 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:40.776772 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 (DEFAULT_TABLE table=default.Table_817b839a30804e609f4916cf44d4dafd [id=95a00c60dd054e6fac7bcd616a3d82b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.779522 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 (DEFAULT_TABLE table=default.Table_817b839a30804e609f4916cf44d4dafd [id=95a00c60dd054e6fac7bcd616a3d82b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.785148 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2ce21e1d654e4989e6459b1edcb6a9 (DEFAULT_TABLE table=default.Table_5faf42ddb1754ee3b8a4f879de32cd43 [id=f6f73c4b50264e5dab2684fd4dfc348b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.785311 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b8287b45fa0b492ca82c22164f3a7837
I20240503 02:12:40.785596 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2ce21e1d654e4989e6459b1edcb6a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.789070 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bb6f5087475411ea280a72ec948ad31 consensus state has no leader
I20240503 02:12:40.790201 4491 tablet_service.cc:1467] Processing CreateTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 (DEFAULT_TABLE table=default.Table_f88ecd4e6c544f43881964c3c4c128d9 [id=f4a85aef4c484cdcbf0079bbbe5803da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.790618 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4940fd3c16424eaaaef59ab50a04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.791996 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet a4a16052d43d45aeb405c5de5dafa041 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:40.794142 4785 tablet_service.cc:1467] Processing CreateTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 (DEFAULT_TABLE table=default.Table_f88ecd4e6c544f43881964c3c4c128d9 [id=f4a85aef4c484cdcbf0079bbbe5803da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.794580 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4940fd3c16424eaaaef59ab50a04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.794751 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a48d5eac44264899a7a372b9f85fb8f5
I20240503 02:12:40.797605 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet b8287b45fa0b492ca82c22164f3a7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.798550 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet b8287b45fa0b492ca82c22164f3a7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.799374 4778 tablet_service.cc:1467] Processing CreateTablet for tablet e08fd23fe2ad43b0a406a9933dac2229 (DEFAULT_TABLE table=default.Table_ba2656ababd74dbeb917e32512185c57 [id=f91a01e310bc43db80553578022bb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.799789 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08fd23fe2ad43b0a406a9933dac2229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.799791 4493 tablet_service.cc:1467] Processing CreateTablet for tablet fd1077af75e848279e5518977c2c1a87 (DEFAULT_TABLE table=default.Table_c012528450084a4498e1aafcccea12c5 [id=e2be94fb2e9a40779c392900c80b8341]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.800200 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1077af75e848279e5518977c2c1a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.804251 5105 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.805004 4788 tablet_service.cc:1467] Processing CreateTablet for tablet fd1077af75e848279e5518977c2c1a87 (DEFAULT_TABLE table=default.Table_c012528450084a4498e1aafcccea12c5 [id=e2be94fb2e9a40779c392900c80b8341]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.805084 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 03dbce232d314f5292ab2410643408af (DEFAULT_TABLE table=default.Table_5693354898d34e2285341bff83a126fd [id=ebd63c60ed8743d8abd1614c7677dff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.805308 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 03dbce232d314f5292ab2410643408af (DEFAULT_TABLE table=default.Table_5693354898d34e2285341bff83a126fd [id=ebd63c60ed8743d8abd1614c7677dff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.805483 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dbce232d314f5292ab2410643408af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.805411 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4dabd6e05948b98f02515023e2aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.806318 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet b8287b45fa0b492ca82c22164f3a7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:40.806674 4650 tablet_service.cc:1467] Processing CreateTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 (DEFAULT_TABLE table=default.Table_f88ecd4e6c544f43881964c3c4c128d9 [id=f4a85aef4c484cdcbf0079bbbe5803da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.806694 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2ce21e1d654e4989e6459b1edcb6a9 (DEFAULT_TABLE table=default.Table_5faf42ddb1754ee3b8a4f879de32cd43 [id=f6f73c4b50264e5dab2684fd4dfc348b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.807009 4647 tablet_service.cc:1467] Processing CreateTablet for tablet e08fd23fe2ad43b0a406a9933dac2229 (DEFAULT_TABLE table=default.Table_ba2656ababd74dbeb917e32512185c57 [id=f91a01e310bc43db80553578022bb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.807291 4644 tablet_service.cc:1467] Processing CreateTablet for tablet fd1077af75e848279e5518977c2c1a87 (DEFAULT_TABLE table=default.Table_c012528450084a4498e1aafcccea12c5 [id=e2be94fb2e9a40779c392900c80b8341]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.807544 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dbce232d314f5292ab2410643408af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.809213 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1077af75e848279e5518977c2c1a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.810289 5105 tablet_bootstrap.cc:654] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.813052 6262 raft_consensus.cc:482] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:40.813205 6262 raft_consensus.cc:504] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:40.813963 6262 leader_election.cc:288] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.816362 4269 catalog_manager.cc:5484] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:40.817073 6262 raft_consensus.cc:482] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:40.817203 6262 raft_consensus.cc:504] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:40.817838 6262 leader_election.cc:288] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.819468 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:40.820158 4494 tablet_service.cc:1467] Processing CreateTablet for tablet e08fd23fe2ad43b0a406a9933dac2229 (DEFAULT_TABLE table=default.Table_ba2656ababd74dbeb917e32512185c57 [id=f91a01e310bc43db80553578022bb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.820540 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08fd23fe2ad43b0a406a9933dac2229. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.821394 4608 leader_election.cc:341] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 53017352b8f34955b116bbfd32f04854
I20240503 02:12:40.821800 5103 tablet_bootstrap.cc:492] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:40.822106 5115 tablet_replica.cc:330] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:40.822345 5115 raft_consensus.cc:2229] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:40.822538 5115 raft_consensus.cc:2258] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:40.823153 5115 ts_tablet_manager.cc:1875] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:40.823535 4608 leader_election.cc:341] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: b161628065704c798020edbe098844c2
I20240503 02:12:40.824968 5103 ts_tablet_manager.cc:1367] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.222s user 0.005s sys 0.010s
I20240503 02:12:40.829929 6560 raft_consensus.cc:482] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.830081 6560 raft_consensus.cc:504] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:40.830224 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6ded21f98c38492387fbc294230e6de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:40.830736 6560 leader_election.cc:288] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:40.830801 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ded21f98c38492387fbc294230e6de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.831833 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f042afb954daa923502ef12377bd5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:40.832046 4675 raft_consensus.cc:2454] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:40.832806 4462 leader_election.cc:302] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.833108 6448 raft_consensus.cc:2781] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.833186 6448 raft_consensus.cc:482] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.833262 6448 raft_consensus.cc:3037] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.833124 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5303be0a72be4225aeced423b445783c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.833688 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5303be0a72be4225aeced423b445783c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.835414 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c3878b4f48444a1cad1c1a9eae99dd33" } new_table_name: "default.Table_0187a6fc086f4da1a49978fae202c36c" modify_external_catalogs: true
I20240503 02:12:40.835537 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_55f57cbbc07a4e969857fcb1074dfa54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:40.835825 6448 raft_consensus.cc:504] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
W20240503 02:12:40.836040 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55f57cbbc07a4e969857fcb1074dfa54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.836478 6448 leader_election.cc:288] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:40.835414 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_896028d24de641e4b765e2a64b27186c" } new_table_name: "default.Table_50a1e12d05be47b5839372a515fe18e0" modify_external_catalogs: true
I20240503 02:12:40.837426 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39f042afb954daa923502ef12377bd5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:40.837625 4675 raft_consensus.cc:3037] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.840302 4675 raft_consensus.cc:2454] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:40.840344 6262 raft_consensus.cc:482] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.840489 6262 raft_consensus.cc:504] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:40.840934 4462 leader_election.cc:302] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:40.841169 6262 leader_election.cc:288] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.841312 6448 raft_consensus.cc:2781] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.841550 6448 raft_consensus.cc:686] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:40.841814 6448 consensus_queue.cc:237] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
W20240503 02:12:40.842357 4608 leader_election.cc:341] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b161628065704c798020edbe098844c2
I20240503 02:12:40.842481 4608 leader_election.cc:302] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:40.842638 4608 raft_consensus.cc:2670] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:40.842991 4608 leader_election.cc:341] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 53017352b8f34955b116bbfd32f04854
I20240503 02:12:40.843178 4608 leader_election.cc:302] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.843562 5103 raft_consensus.cc:348] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.845754 5103 raft_consensus.cc:374] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.851089 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 8425bb6a789a4aaf8c6e3714943852cc (DEFAULT_TABLE table=default.Table_bc90b6bcb7514fbcace5a7639189ae7c [id=e6c829aba6624ff2a7b4c05f720d71d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.851298 5105 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.851438 4778 tablet_service.cc:1467] Processing CreateTablet for tablet b9f9fa6388de4fc2a607adede400cdca (DEFAULT_TABLE table=default.Table_4284eccc6a0f475bbfe7d8a92029b257 [id=2602b62873a44f6c8d48770dfd70518c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.851537 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425bb6a789a4aaf8c6e3714943852cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.851507 5103 raft_consensus.cc:729] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.851775 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f9fa6388de4fc2a607adede400cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.851837 4500 tablet_service.cc:1467] Processing CreateTablet for tablet f8be9e5380b4466baccd16483e940588 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.852078 4496 tablet_service.cc:1467] Processing CreateTablet for tablet b9f9fa6388de4fc2a607adede400cdca (DEFAULT_TABLE table=default.Table_4284eccc6a0f475bbfe7d8a92029b257 [id=2602b62873a44f6c8d48770dfd70518c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.852185 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be9e5380b4466baccd16483e940588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.852432 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f9fa6388de4fc2a607adede400cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.855661 4788 tablet_service.cc:1467] Processing CreateTablet for tablet b82046e9fc994799830215c011367503 (DEFAULT_TABLE table=default.Table_c0df938d7e494f96a93c3c23baa3bb09 [id=9d8f736388d543fdb294d467cb46d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.856012 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82046e9fc994799830215c011367503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.857558 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:40.858355 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:40.858850 4778 tablet_service.cc:1467] Processing CreateTablet for tablet f8be9e5380b4466baccd16483e940588 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.851444 5105 ts_tablet_manager.cc:1367] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.001s
I20240503 02:12:40.859218 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be9e5380b4466baccd16483e940588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.859333 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 8c90bb18baae414d8d9a020cd38196fb (DEFAULT_TABLE table=default.Table_196b58d9443a45dba25db7d08ee04d20 [id=47012a8a78e940f386aec7c2eb91b88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.858559 5103 consensus_queue.cc:260] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.859779 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c90bb18baae414d8d9a020cd38196fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.863147 5103 ts_tablet_manager.cc:1398] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.037s user 0.002s sys 0.001s
I20240503 02:12:40.865574 5103 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:40.870723 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2ce21e1d654e4989e6459b1edcb6a9 (DEFAULT_TABLE table=default.Table_5faf42ddb1754ee3b8a4f879de32cd43 [id=f6f73c4b50264e5dab2684fd4dfc348b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.871170 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2ce21e1d654e4989e6459b1edcb6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.864849 5105 raft_consensus.cc:348] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.871593 5105 raft_consensus.cc:374] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.872184 5105 raft_consensus.cc:729] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.873957 5105 consensus_queue.cc:260] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.879004 5105 ts_tablet_manager.cc:1398] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240503 02:12:40.880829 4778 tablet_service.cc:1467] Processing CreateTablet for tablet a3c6e640f3c04b6a96493dadcfb35680 (DEFAULT_TABLE table=default.Table_27706a397e30464e8687aacf693fc733 [id=d8b4a3c1f25c43fdbf01a83844c69693]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.881263 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c6e640f3c04b6a96493dadcfb35680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.881573 5105 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.882854 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 780fdcea52484fa3b6b30fcee001ba2b (DEFAULT_TABLE table=default.Table_38a612f7f6034cf28b8329d12c6f97ac [id=79a7ea79db094d258775723e69c71ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.883224 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780fdcea52484fa3b6b30fcee001ba2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.884770 4789 tablet_service.cc:1467] Processing CreateTablet for tablet b565a19d7e444065aae5b9e5604d36fa (DEFAULT_TABLE table=default.Table_9c84ec1ce10e42188f751afc07da15f0 [id=265ecab87a494c4eb410f75a226a1ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.884666 5103 tablet_bootstrap.cc:654] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.885151 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b565a19d7e444065aae5b9e5604d36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.889250 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 166599de742e42ea8733d46ead986341 (DEFAULT_TABLE table=default.Table_b48ee0fa4b364d7ca10d28cc8224840e [id=3cce450e263f4c38bff14b82f72f8470]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:40.889447 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 0bb6f5087475411ea280a72ec948ad31 (table default.Table_d199ea94999f4b009ecca81b9c654f93 [id=4b238c1a9bcb4638844c6aa478082c43], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0bb6f5087475411ea280a72ec948ad31 consensus state has no leader
I20240503 02:12:40.889611 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166599de742e42ea8733d46ead986341. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:40.891258 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 48268.3us
I20240503 02:12:40.891412 4497 tablet_service.cc:1467] Processing CreateTablet for tablet a3c6e640f3c04b6a96493dadcfb35680 (DEFAULT_TABLE table=default.Table_27706a397e30464e8687aacf693fc733 [id=d8b4a3c1f25c43fdbf01a83844c69693]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:40.891651 4608 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:40.891814 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c6e640f3c04b6a96493dadcfb35680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.892408 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 03dbce232d314f5292ab2410643408af (DEFAULT_TABLE table=default.Table_5693354898d34e2285341bff83a126fd [id=ebd63c60ed8743d8abd1614c7677dff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.892448 4648 tablet_service.cc:1467] Processing CreateTablet for tablet b82046e9fc994799830215c011367503 (DEFAULT_TABLE table=default.Table_c0df938d7e494f96a93c3c23baa3bb09 [id=9d8f736388d543fdb294d467cb46d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.892591 4637 tablet_service.cc:1467] Processing CreateTablet for tablet b9f9fa6388de4fc2a607adede400cdca (DEFAULT_TABLE table=default.Table_4284eccc6a0f475bbfe7d8a92029b257 [id=2602b62873a44f6c8d48770dfd70518c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.893095 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 8425bb6a789a4aaf8c6e3714943852cc (DEFAULT_TABLE table=default.Table_bc90b6bcb7514fbcace5a7639189ae7c [id=e6c829aba6624ff2a7b4c05f720d71d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.893330 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:40.893522 4608 leader_election.cc:341] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: d271d2c112ba44b68d2933a6ebfe7a18
I20240503 02:12:40.893620 4608 leader_election.cc:302] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:40.893638 4640 tablet_service.cc:1467] Processing CreateTablet for tablet f8be9e5380b4466baccd16483e940588 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.893656 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 780fdcea52484fa3b6b30fcee001ba2b (DEFAULT_TABLE table=default.Table_38a612f7f6034cf28b8329d12c6f97ac [id=79a7ea79db094d258775723e69c71ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.893846 6374 raft_consensus.cc:2726] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:40.894006 6374 raft_consensus.cc:2726] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:40.893962 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 8c90bb18baae414d8d9a020cd38196fb (DEFAULT_TABLE table=default.Table_196b58d9443a45dba25db7d08ee04d20 [id=47012a8a78e940f386aec7c2eb91b88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.894064 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780fdcea52484fa3b6b30fcee001ba2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.894318 4652 tablet_service.cc:1467] Processing CreateTablet for tablet a3c6e640f3c04b6a96493dadcfb35680 (DEFAULT_TABLE table=default.Table_27706a397e30464e8687aacf693fc733 [id=d8b4a3c1f25c43fdbf01a83844c69693]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.894649 4639 tablet_service.cc:1467] Processing CreateTablet for tablet b565a19d7e444065aae5b9e5604d36fa (DEFAULT_TABLE table=default.Table_9c84ec1ce10e42188f751afc07da15f0 [id=265ecab87a494c4eb410f75a226a1ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.894940 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 166599de742e42ea8733d46ead986341 (DEFAULT_TABLE table=default.Table_b48ee0fa4b364d7ca10d28cc8224840e [id=3cce450e263f4c38bff14b82f72f8470]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.895251 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 780fdcea52484fa3b6b30fcee001ba2b (DEFAULT_TABLE table=default.Table_38a612f7f6034cf28b8329d12c6f97ac [id=79a7ea79db094d258775723e69c71ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.896210 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:40.897632 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_001ef146e7b442e99269b86b5b32c214" } new_table_name: "default.Table_dddd71becaf8450f999400d322914922" modify_external_catalogs: true
I20240503 02:12:40.898746 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b591bdbd9f5244d1990cce7fe667b7a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:40.899279 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b591bdbd9f5244d1990cce7fe667b7a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:40.901551 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 166599de742e42ea8733d46ead986341 (DEFAULT_TABLE table=default.Table_b48ee0fa4b364d7ca10d28cc8224840e [id=3cce450e263f4c38bff14b82f72f8470]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.903270 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 8c90bb18baae414d8d9a020cd38196fb (DEFAULT_TABLE table=default.Table_196b58d9443a45dba25db7d08ee04d20 [id=47012a8a78e940f386aec7c2eb91b88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.906522 5105 tablet_bootstrap.cc:654] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.902478 4494 tablet_service.cc:1467] Processing CreateTablet for tablet b82046e9fc994799830215c011367503 (DEFAULT_TABLE table=default.Table_c0df938d7e494f96a93c3c23baa3bb09 [id=9d8f736388d543fdb294d467cb46d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.911446 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c90bb18baae414d8d9a020cd38196fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.916240 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 8425bb6a789a4aaf8c6e3714943852cc (DEFAULT_TABLE table=default.Table_bc90b6bcb7514fbcace5a7639189ae7c [id=e6c829aba6624ff2a7b4c05f720d71d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.916641 5105 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:40.917909 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425bb6a789a4aaf8c6e3714943852cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.917940 5105 ts_tablet_manager.cc:1367] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.001s
I20240503 02:12:40.918689 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82046e9fc994799830215c011367503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.923573 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4940fd3c16424eaaaef59ab50a04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.925240 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad6a7ea9a0740b9b781985d5cc7d25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.925504 5105 raft_consensus.cc:348] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.927985 5105 raft_consensus.cc:374] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.929140 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4dabd6e05948b98f02515023e2aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.928613 5105 raft_consensus.cc:729] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.933178 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2ce21e1d654e4989e6459b1edcb6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.931856 5105 consensus_queue.cc:260] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.937085 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08fd23fe2ad43b0a406a9933dac2229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.937592 5105 ts_tablet_manager.cc:1398] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240503 02:12:40.937901 5105 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:40.939275 5105 tablet_bootstrap.cc:654] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.941179 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1077af75e848279e5518977c2c1a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.944674 5106 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:40.946499 5106 tablet_bootstrap.cc:654] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.947604 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dbce232d314f5292ab2410643408af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.948951 6296 raft_consensus.cc:482] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.949170 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82046e9fc994799830215c011367503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.949088 6296 raft_consensus.cc:504] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.949736 6296 leader_election.cc:288] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.953356 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f9fa6388de4fc2a607adede400cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.953598 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c90bb18baae414d8d9a020cd38196fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.957342 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be9e5380b4466baccd16483e940588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.957823 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425bb6a789a4aaf8c6e3714943852cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.960036 6478 consensus_queue.cc:1035] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.965368 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b565a19d7e444065aae5b9e5604d36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.965576 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166599de742e42ea8733d46ead986341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.965808 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780fdcea52484fa3b6b30fcee001ba2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.969110 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c6e640f3c04b6a96493dadcfb35680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.983814 6349 raft_consensus.cc:482] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.984032 6349 raft_consensus.cc:504] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.985147 6349 leader_election.cc:288] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:40.997598 6262 raft_consensus.cc:482] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.997797 6262 raft_consensus.cc:504] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:40.998404 6262 leader_election.cc:288] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:40.999104 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:40.999358 4813 raft_consensus.cc:2454] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:40.999557 4491 tablet_service.cc:1467] Processing CreateTablet for tablet b565a19d7e444065aae5b9e5604d36fa (DEFAULT_TABLE table=default.Table_9c84ec1ce10e42188f751afc07da15f0 [id=265ecab87a494c4eb410f75a226a1ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.999946 4608 leader_election.cc:302] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:41.000312 6262 raft_consensus.cc:2781] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.000380 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166599de742e42ea8733d46ead986341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.000416 6262 raft_consensus.cc:482] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.000491 6262 raft_consensus.cc:3037] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.000819 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 774) took 1359 ms. Trace:
I20240503 02:12:41.000897 5159 rpcz_store.cc:276] 0503 02:12:39.641340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.641405 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:40.776287 (+1134882us) ts_tablet_manager.cc:1120] Deleting tablet 4ced4dfb2cf541af98d50d22d1a0f0ac
0503 02:12:41.000704 (+224417us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.000801 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134732}
I20240503 02:12:41.001261 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 775) took 1359 ms. Trace:
I20240503 02:12:41.001324 5159 rpcz_store.cc:276] 0503 02:12:39.641794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.641837 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:41.001182 (+1359345us) ts_tablet_manager.cc:1120] Deleting tablet ea7afb429d64497abcd9f1d0910651c3
0503 02:12:41.001192 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.001248 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359209}
I20240503 02:12:41.001538 5159 tablet_replica.cc:330] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:41.001703 5159 raft_consensus.cc:2229] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.001832 5159 raft_consensus.cc:2258] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.002341 5159 ts_tablet_manager.cc:1875] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.002955 6262 raft_consensus.cc:504] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.005216 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b565a19d7e444065aae5b9e5604d36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.005326 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:41.005510 4813 raft_consensus.cc:3037] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.005453 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:40.763029 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 703) took 2488 ms. Trace:
I20240503 02:12:41.006361 5114 rpcz_store.cc:276] 0503 02:12:38.274729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.275220 (+ 491us) service_pool.cc:225] Handling call
0503 02:12:40.586016 (+2310796us) ts_tablet_manager.cc:1120] Deleting tablet 2152f205cf1b4ec7ae03605bbfa007e1
0503 02:12:40.586031 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.605503 (+ 19472us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.608448 (+ 2945us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:40.763006 (+154558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310567}
I20240503 02:12:41.006701 5114 tablet_replica.cc:330] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.006865 5114 raft_consensus.cc:2229] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:41.007138 5114 raft_consensus.cc:2258] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.007700 5114 ts_tablet_manager.cc:1875] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.008055 4813 raft_consensus.cc:2454] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.008510 4608 leader_election.cc:302] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:41.008864 6374 raft_consensus.cc:2781] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.009002 6374 raft_consensus.cc:686] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.009244 6374 consensus_queue.cc:237] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.009945 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:41.010625 4608 leader_election.cc:341] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.011032 6262 leader_election.cc:288] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:41.011874 4608 leader_election.cc:341] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:41.026976 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 135891us
I20240503 02:12:41.040192 6262 raft_consensus.cc:482] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.040380 6262 raft_consensus.cc:504] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.041201 6262 leader_election.cc:288] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.041895 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4225c1bc133d4e18a4a47fd58820ca10" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:41.042294 4608 leader_election.cc:341] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 4225c1bc133d4e18a4a47fd58820ca10
W20240503 02:12:41.042501 4608 leader_election.cc:341] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.042578 4608 leader_election.cc:302] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.043025 6262 raft_consensus.cc:2726] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.055903 5115 ts_tablet_manager.cc:1888] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.056061 5115 log.cc:1196] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b161628065704c798020edbe098844c2
I20240503 02:12:41.056494 5115 ts_tablet_manager.cc:1909] T b161628065704c798020edbe098844c2 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:41.056883 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 672) took 3981 ms. Trace:
I20240503 02:12:41.057006 5115 rpcz_store.cc:276] 0503 02:12:37.075142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.080869 (+ 5727us) service_pool.cc:225] Handling call
0503 02:12:40.637916 (+3557047us) ts_tablet_manager.cc:1120] Deleting tablet b161628065704c798020edbe098844c2
0503 02:12:40.821938 (+184022us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:40.847538 (+ 25600us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.055781 (+208243us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.056856 (+ 1075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556846}
I20240503 02:12:41.057390 5115 tablet_replica.cc:330] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:41.057643 5115 raft_consensus.cc:2229] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.057830 5115 raft_consensus.cc:2258] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.058425 5115 ts_tablet_manager.cc:1875] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.060799 5106 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.060969 5106 ts_tablet_manager.cc:1367] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.000s
I20240503 02:12:41.062261 5106 raft_consensus.cc:348] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.062484 5106 raft_consensus.cc:374] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.062547 5106 raft_consensus.cc:729] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.062911 5106 consensus_queue.cc:260] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.063560 5106 ts_tablet_manager.cc:1398] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:41.063899 5106 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:41.063902 5103 tablet_bootstrap.cc:492] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.064049 5103 ts_tablet_manager.cc:1367] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.199s user 0.002s sys 0.018s
I20240503 02:12:41.065541 5106 tablet_bootstrap.cc:654] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.073094 4746 leader_election.cc:302] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:41.076654 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4ced4dfb2cf541af98d50d22d1a0f0ac because the tablet was not found. No further retry: Not found: Tablet not found: 4ced4dfb2cf541af98d50d22d1a0f0ac
I20240503 02:12:41.077040 6349 raft_consensus.cc:2726] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.077118 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ea7afb429d64497abcd9f1d0910651c3 because the tablet was not found. No further retry: Not found: Tablet not found: ea7afb429d64497abcd9f1d0910651c3
W20240503 02:12:41.077409 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 345479us
W20240503 02:12:41.077674 4746 leader_election.cc:341] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.077749 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 2152f205cf1b4ec7ae03605bbfa007e1 (table default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]) successfully deleted
I20240503 02:12:41.078482 4746 leader_election.cc:302] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.080688 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b161628065704c798020edbe098844c2 (table default.Table_dacc67e3ff0c49fc9dddc714ec081269 [id=ea285244a0254bf4aa6cc9cf4de92a74]) successfully deleted
I20240503 02:12:41.082618 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_38a612f7f6034cf28b8329d12c6f97ac" } new_table_name: "default.Table_36781fb48fda4b8dabff9356f8dfc52f" modify_external_catalogs: true
W20240503 02:12:41.083979 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b143c398ccbe42f09fa2e33f04f0d4d7 consensus state has no leader
W20240503 02:12:41.084225 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f630bf6a5740758fa1f81477ec59d3 consensus state has no leader
I20240503 02:12:41.084336 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b9f90c9e6ae3489b9809bed574c04644" } modify_external_catalogs: true
I20240503 02:12:41.084529 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6d070a69982c45b7904a12c6df0386b6" } modify_external_catalogs: true
I20240503 02:12:41.084573 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b9f90c9e6ae3489b9809bed574c04644" } modify_external_catalogs: true
I20240503 02:12:41.084703 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6d070a69982c45b7904a12c6df0386b6" } modify_external_catalogs: true
I20240503 02:12:41.085342 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4a15903f49d74e708bd42ab681cb77a5" } modify_external_catalogs: true
I20240503 02:12:41.085539 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4a15903f49d74e708bd42ab681cb77a5" } modify_external_catalogs: true
W20240503 02:12:41.085884 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d31e874e285c4b20b4547048a3150d4a consensus state has no leader
I20240503 02:12:41.086004 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f9386d5ccb884c81a0146719976d0b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:41.086198 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b" } modify_external_catalogs: true
I20240503 02:12:41.086340 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_63fd66b7e8434aa5a8f50cf8cee0c89b" } modify_external_catalogs: true
W20240503 02:12:41.086709 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9386d5ccb884c81a0146719976d0b36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.091686 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_958a3d4a699a4387b0ff4c721a958d54 [id=ad0169357ed4491c92e35a64d6a202bb] alter complete (version 1)
I20240503 02:12:41.083982 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8e334557d1ca4d7faa91def038f0afc4" } modify_external_catalogs: true
I20240503 02:12:41.093220 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8e334557d1ca4d7faa91def038f0afc4" } modify_external_catalogs: true
I20240503 02:12:41.095268 4269 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_1d13d28ae8bc4af08a1b447dc0e62c6b [id=ae1d450680154e00bb20d21e106dbd53] alter complete (version 3)
I20240503 02:12:41.096463 6349 raft_consensus.cc:2726] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.102542 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 780fdcea52484fa3b6b30fcee001ba2b consensus state has no leader
I20240503 02:12:41.102999 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_66c2a4cc5ab648c5938e6aecd6792779" } new_table_name: "default.Table_28a6ed33b1844f33bad1663e980b3419" modify_external_catalogs: true
I20240503 02:12:41.103049 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0b8cca9f2a7641e6b27283f7a08f0dfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.103624 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b8cca9f2a7641e6b27283f7a08f0dfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.106699 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c13f257eae44689af07cc31ea254c4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.106818 6349 raft_consensus.cc:482] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.106954 6349 raft_consensus.cc:504] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.107569 6349 leader_election.cc:288] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:41.107548 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.107759 6349 raft_consensus.cc:482] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.107877 6349 raft_consensus.cc:504] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.108124 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_2e0422a1f16245238c15c03daefcbe32 [id=eff7b93c47234c2fa5a3d6ecb95c5cae] alter complete (version 4)
I20240503 02:12:41.108438 6349 leader_election.cc:288] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:41.109475 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_afc821e2c5794ea190f4aec89a0f4676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.110057 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afc821e2c5794ea190f4aec89a0f4676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.113564 5106 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.113626 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.114078 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.114565 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.114637 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5468ef29b1e94242b3bb4d5dadf22680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.115224 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5468ef29b1e94242b3bb4d5dadf22680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:41.116537 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad383b84ad1646108cc54f1b7ef148ad consensus state has no leader
I20240503 02:12:41.117158 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_742dacc5e26a41fe86b410fe0cfb5b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.117733 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_742dacc5e26a41fe86b410fe0cfb5b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.114099 5106 ts_tablet_manager.cc:1367] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240503 02:12:41.119709 5106 raft_consensus.cc:348] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.120155 5106 raft_consensus.cc:374] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.120126 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:41.120494 5106 raft_consensus.cc:729] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:41.120723 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.121426 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.121081 5106 consensus_queue.cc:260] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.122007 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 95681e67db1e4485876d530133e8a514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.122522 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.125310 5106 ts_tablet_manager.cc:1398] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:41.126633 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9994ca5aea48d3bc25c097dddba3c3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.126802 5106 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:41.127288 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.127509 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce11c70a924b8d92d8f36fc7401dfd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.128000 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.128468 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.128968 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:41.129949 4746 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:41.130208 4746 leader_election.cc:341] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:41.130450 4746 leader_election.cc:341] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:41.130643 4746 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.133800 5106 tablet_bootstrap.cc:654] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.076536 5159 ts_tablet_manager.cc:1888] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.136708 5159 log.cc:1196] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/48e713ee37d042309166b0f118e14986
I20240503 02:12:41.137195 5159 ts_tablet_manager.cc:1909] T 48e713ee37d042309166b0f118e14986 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:41.138206 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 776) took 1495 ms. Trace:
I20240503 02:12:41.138346 5159 rpcz_store.cc:276] 0503 02:12:39.642410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.642501 (+ 91us) service_pool.cc:225] Handling call
0503 02:12:41.001498 (+1358997us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:41.001505 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.073375 (+ 71870us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.076425 (+ 3050us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.138115 (+ 61690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358870}
I20240503 02:12:41.138973 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 778) took 1494 ms. Trace:
I20240503 02:12:41.140798 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 48e713ee37d042309166b0f118e14986 (table default.Table_fb11bd168d9b4a22940b384400a5c6d5 [id=b4c6a45a47e647b1947d0d7f2ea30afa]) successfully deleted
I20240503 02:12:41.139714 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.140202 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.141184 5159 rpcz_store.cc:276] 0503 02:12:39.644051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.644094 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:41.138865 (+1494771us) ts_tablet_manager.cc:1120] Deleting tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
0503 02:12:41.138876 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.138957 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494643}
I20240503 02:12:41.141954 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 771) took 1502 ms. Trace:
I20240503 02:12:41.142329 5159 rpcz_store.cc:276] 0503 02:12:39.639637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.644038 (+ 4401us) service_pool.cc:225] Handling call
0503 02:12:41.141848 (+1497810us) ts_tablet_manager.cc:1120] Deleting tablet b91abaf51abb4c2ca65412c3771c1ef2
0503 02:12:41.141857 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.141935 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497620}
I20240503 02:12:41.142576 6317 raft_consensus.cc:686] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
W20240503 02:12:41.142845 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 because the tablet was not found. No further retry: Not found: Tablet not found: 1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:41.142200 5105 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.143450 5105 ts_tablet_manager.cc:1367] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.206s user 0.005s sys 0.005s
I20240503 02:12:41.143573 5103 raft_consensus.cc:348] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.144155 5105 raft_consensus.cc:348] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.144212 6317 consensus_queue.cc:237] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:41.144388 5105 raft_consensus.cc:374] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.144455 5105 raft_consensus.cc:729] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.144690 5105 consensus_queue.cc:260] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.145246 5105 ts_tablet_manager.cc:1398] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:41.145552 5105 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.147014 5105 tablet_bootstrap.cc:654] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.108995 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 59855ab3e42b4e019b769007ccbbc360 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.109706 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.110124 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178760c21904207b6c8317093871f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.110507 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e713ee37d042309166b0f118e14986 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.147912 5103 raft_consensus.cc:374] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.148617 5103 raft_consensus.cc:729] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.149060 5103 consensus_queue.cc:260] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:41.153792 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b91abaf51abb4c2ca65412c3771c1ef2 because the tablet was not found. No further retry: Not found: Tablet not found: b91abaf51abb4c2ca65412c3771c1ef2
I20240503 02:12:41.147883 5159 tablet_replica.cc:330] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:41.152560 6349 raft_consensus.cc:482] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.156102 6349 raft_consensus.cc:504] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.156776 6349 leader_election.cc:288] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.157202 5159 raft_consensus.cc:2229] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.157456 5159 raft_consensus.cc:2258] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.158082 5159 ts_tablet_manager.cc:1875] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.158114 5106 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.158442 5106 ts_tablet_manager.cc:1367] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.000s
I20240503 02:12:41.160117 5106 raft_consensus.cc:348] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.160645 5106 raft_consensus.cc:374] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.160853 5106 raft_consensus.cc:729] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.161389 5106 consensus_queue.cc:260] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.162739 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet 3c421694ce1e424da7953d061a6bca3e from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:41.163551 5106 ts_tablet_manager.cc:1398] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:41.164800 5106 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:41.129372 4462 consensus_peers.cc:485] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41d44fe8f6334572996665bdec60e2e5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.146029 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce11c70a924b8d92d8f36fc7401dfd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:41.171752 4462 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d39f042afb954daa923502ef12377bd5
I20240503 02:12:41.146001 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.146001 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c13f257eae44689af07cc31ea254c4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:41.172194 4462 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d39f042afb954daa923502ef12377bd5
I20240503 02:12:41.173779 5106 tablet_bootstrap.cc:654] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.150689 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9994ca5aea48d3bc25c097dddba3c3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.111776 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet d39f042afb954daa923502ef12377bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.182143 5103 ts_tablet_manager.cc:1398] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.039s user 0.001s sys 0.000s
I20240503 02:12:41.183717 6374 raft_consensus.cc:482] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.183933 6374 raft_consensus.cc:504] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.185307 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:41.185534 4811 raft_consensus.cc:2454] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.185909 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.186038 4608 leader_election.cc:302] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:41.186275 6374 leader_election.cc:288] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.186401 6374 raft_consensus.cc:2781] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.186473 6374 raft_consensus.cc:482] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.186537 6374 raft_consensus.cc:3037] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:41.186908 4608 leader_election.cc:341] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.189278 5103 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.192525 5159 ts_tablet_manager.cc:1888] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.192659 5159 log.cc:1196] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7461aa76a1af41cb9253285bda708f30
I20240503 02:12:41.194479 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c9090b11c8a4bf0b4b92523b9431b79 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:41.101585 5114 ts_tablet_manager.cc:1888] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.195873 5114 log.cc:1196] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/41ef3c6e5e7d40249b112144975ba220
I20240503 02:12:41.196462 5114 ts_tablet_manager.cc:1909] T 41ef3c6e5e7d40249b112144975ba220 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.202785 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0c3b9a888b894897b7cc3cf08402febd
I20240503 02:12:41.202821 6374 raft_consensus.cc:504] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.204543 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:41.205587 4608 leader_election.cc:341] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.206037 4272 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:41.206310 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5e4db5204e4acf8719100d02d533fb" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:41.206521 4811 raft_consensus.cc:3037] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.209165 4811 raft_consensus.cc:2454] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.209280 6561 raft_consensus.cc:482] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.209399 6561 raft_consensus.cc:504] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.209619 6374 leader_election.cc:288] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.209968 6561 leader_election.cc:288] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.210261 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3b9a888b894897b7cc3cf08402febd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.210485 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3b9a888b894897b7cc3cf08402febd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.210791 4608 leader_election.cc:302] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.211361 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3b9a888b894897b7cc3cf08402febd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.212057 6561 raft_consensus.cc:2781] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.212186 6561 raft_consensus.cc:686] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.212455 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 28c1a81589df496b92b980353f00e594
I20240503 02:12:41.212425 6561 consensus_queue.cc:237] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.212829 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.213080 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:41.213253 4813 raft_consensus.cc:2454] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.213495 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.213768 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.213948 4529 raft_consensus.cc:2454] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.215071 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet efd8e0353b5f4ce484ac81d646b5323d
I20240503 02:12:41.215605 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 39c48b14bc624c28a871d45e9429ee30
I20240503 02:12:41.213200 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5f8656a886564f06931039b516162b51
I20240503 02:12:41.216150 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c1a81589df496b92b980353f00e594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.216646 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3c421694ce1e424da7953d061a6bca3e
I20240503 02:12:41.216683 4608 leader_election.cc:302] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:41.217419 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet efd8e0353b5f4ce484ac81d646b5323d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.217610 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c1a81589df496b92b980353f00e594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.219754 4267 catalog_manager.cc:6554] ReplaceTablet: tablet 3c421694ce1e424da7953d061a6bca3e deleted and replaced by tablet 03fc31800276415fa68d95339025a55f
I20240503 02:12:41.221170 6374 raft_consensus.cc:2781] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.221362 6374 raft_consensus.cc:482] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.221527 6374 raft_consensus.cc:3037] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.223994 5159 ts_tablet_manager.cc:1909] T 7461aa76a1af41cb9253285bda708f30 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:41.225267 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:41.204890 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 705) took 2929 ms. Trace:
I20240503 02:12:41.226719 5114 rpcz_store.cc:276] 0503 02:12:38.274957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.275481 (+ 524us) service_pool.cc:225] Handling call
0503 02:12:41.006657 (+2731176us) ts_tablet_manager.cc:1120] Deleting tablet 41ef3c6e5e7d40249b112144975ba220
0503 02:12:41.006666 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.043820 (+ 37154us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.101459 (+ 57639us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.204874 (+103415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731035}
I20240503 02:12:41.226879 6317 consensus_queue.cc:1035] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.227355 5114 tablet_replica.cc:330] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.227602 5114 raft_consensus.cc:2229] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.227887 5114 raft_consensus.cc:2258] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.228504 5114 ts_tablet_manager.cc:1875] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.229728 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8656a886564f06931039b516162b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.230628 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 9003cc376aa149b59947bc32bd071163 (DEFAULT_TABLE table=default.Table_4b3d4e5854b34468a48715e9093f70e2 [id=5c675fea25ce4e6b87395b51899da369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.231144 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9003cc376aa149b59947bc32bd071163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.232395 5106 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.232702 5106 ts_tablet_manager.cc:1367] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.000s
I20240503 02:12:41.233922 5106 raft_consensus.cc:348] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.234328 5106 raft_consensus.cc:374] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.234454 5106 raft_consensus.cc:729] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.234797 5106 consensus_queue.cc:260] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.235594 5106 ts_tablet_manager.cc:1398] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:41.236719 5106 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:41.238376 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 41d44fe8f6334572996665bdec60e2e5 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:41.240219 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c48b14bc624c28a871d45e9429ee30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.241585 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 41ef3c6e5e7d40249b112144975ba220 (table default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]) successfully deleted
I20240503 02:12:41.241822 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet efd8e0353b5f4ce484ac81d646b5323d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.245052 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c421694ce1e424da7953d061a6bca3e with delete_type TABLET_DATA_DELETED (replaced by tablet 03fc31800276415fa68d95339025a55f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.246757 5103 tablet_bootstrap.cc:654] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.248900 5115 ts_tablet_manager.cc:1888] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.238041 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8656a886564f06931039b516162b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.238618 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c1a81589df496b92b980353f00e594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.251260 5115 log.cc:1196] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7178760c21904207b6c8317093871f63
I20240503 02:12:41.238997 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet efd8e0353b5f4ce484ac81d646b5323d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.239739 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c48b14bc624c28a871d45e9429ee30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.240702 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c421694ce1e424da7953d061a6bca3e with delete_type TABLET_DATA_DELETED (replaced by tablet 03fc31800276415fa68d95339025a55f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.250099 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 9003cc376aa149b59947bc32bd071163 (DEFAULT_TABLE table=default.Table_4b3d4e5854b34468a48715e9093f70e2 [id=5c675fea25ce4e6b87395b51899da369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.251600 5115 ts_tablet_manager.cc:1909] T 7178760c21904207b6c8317093871f63 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:41.251994 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 674) took 4176 ms. Trace:
I20240503 02:12:41.252137 5115 rpcz_store.cc:276] 0503 02:12:37.075363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.081742 (+ 6379us) service_pool.cc:225] Handling call
0503 02:12:41.057338 (+3975596us) ts_tablet_manager.cc:1120] Deleting tablet 7178760c21904207b6c8317093871f63
0503 02:12:41.057349 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.114259 (+ 56910us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.248788 (+134529us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.251974 (+ 3186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3652032}
I20240503 02:12:41.252388 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 067eaec72a314d7a820ffe0adbcb75f8 (DEFAULT_TABLE table=default.Table_ecb8bc8e313042c292129fa02342ed2f [id=341076d5e2eb4c70b1d242be429eff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.252660 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 675) took 4177 ms. Trace:
I20240503 02:12:41.252728 5115 rpcz_store.cc:276] 0503 02:12:37.075465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.144245 (+ 68780us) service_pool.cc:225] Handling call
0503 02:12:41.252529 (+4108284us) ts_tablet_manager.cc:1120] Deleting tablet 5cd296bb3b574d6cbb0252ad10ada00c
0503 02:12:41.252540 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.252644 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3834976}
I20240503 02:12:41.253144 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 676) took 4108 ms. Trace:
I20240503 02:12:41.254582 4789 tablet_service.cc:1467] Processing CreateTablet for tablet ecb2b30119ec43cdabdb2319222569c3 (DEFAULT_TABLE table=default.Table_348455618e27462284c9005cc2368dd7 [id=40fd861bb4f3406689ebe14e7ff2e9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.255626 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.255625 4499 tablet_service.cc:1467] Processing CreateTablet for tablet ecb2b30119ec43cdabdb2319222569c3 (DEFAULT_TABLE table=default.Table_348455618e27462284c9005cc2368dd7 [id=40fd861bb4f3406689ebe14e7ff2e9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.256572 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb2b30119ec43cdabdb2319222569c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:41.262777 4746 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.262861 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 067eaec72a314d7a820ffe0adbcb75f8 (DEFAULT_TABLE table=default.Table_ecb8bc8e313042c292129fa02342ed2f [id=341076d5e2eb4c70b1d242be429eff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.263032 4746 leader_election.cc:302] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.263232 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067eaec72a314d7a820ffe0adbcb75f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.263876 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9003cc376aa149b59947bc32bd071163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.268616 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce96306714144bfb5ee05bbf4d25572 (DEFAULT_TABLE table=default.Table_1a6867e7591e44dc942982feb523e922 [id=4712f930b6984081956e2d9864f8c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.268980 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce96306714144bfb5ee05bbf4d25572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.269127 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067eaec72a314d7a820ffe0adbcb75f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.269369 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb2b30119ec43cdabdb2319222569c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.273067 6560 raft_consensus.cc:482] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.273211 6560 raft_consensus.cc:504] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:41.273800 6560 leader_election.cc:288] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:41.275048 4462 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Not found: Tablet not found: 7178760c21904207b6c8317093871f63
I20240503 02:12:41.255082 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.258761 5106 tablet_bootstrap.cc:654] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:41.278515 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 61337.2us
I20240503 02:12:41.280368 5115 rpcz_store.cc:276] 0503 02:12:37.145197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.145299 (+ 102us) service_pool.cc:225] Handling call
0503 02:12:41.253043 (+4107744us) ts_tablet_manager.cc:1120] Deleting tablet 6d08e7d17f1644749a9a2e7175b3191a
0503 02:12:41.253051 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.253128 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3834247}
I20240503 02:12:41.280973 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 677) took 4135 ms. Trace:
I20240503 02:12:41.281123 5115 rpcz_store.cc:276] 0503 02:12:37.145771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.145811 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:41.280835 (+4135024us) ts_tablet_manager.cc:1120] Deleting tablet 0a8c57ddf3d8428e8fa5ffb061dc054b
0503 02:12:41.280845 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.280956 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3859801}
I20240503 02:12:41.281548 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 696) took 3148 ms. Trace:
I20240503 02:12:41.281474 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce96306714144bfb5ee05bbf4d25572 (DEFAULT_TABLE table=default.Table_1a6867e7591e44dc942982feb523e922 [id=4712f930b6984081956e2d9864f8c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.281754 5115 rpcz_store.cc:276] 0503 02:12:38.132590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.133334 (+ 744us) service_pool.cc:225] Handling call
0503 02:12:41.281444 (+3148110us) ts_tablet_manager.cc:1120] Deleting tablet d84412b28b4c41c2a3ce18702403dcd8
0503 02:12:41.281453 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.281532 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3147687}
I20240503 02:12:41.281860 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce96306714144bfb5ee05bbf4d25572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.282142 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 695) took 3149 ms. Trace:
I20240503 02:12:41.282270 5115 rpcz_store.cc:276] 0503 02:12:38.132458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.132519 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:41.282050 (+3149531us) ts_tablet_manager.cc:1120] Deleting tablet 6b15997a94b74ad3808d965acfb5d155
0503 02:12:41.282058 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.282127 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3147783}
I20240503 02:12:41.283030 6349 raft_consensus.cc:2726] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.283491 4746 leader_election.cc:341] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.283635 4746 leader_election.cc:302] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.283910 6349 raft_consensus.cc:2726] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.284420 4746 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.284528 4746 leader_election.cc:302] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.284716 6349 raft_consensus.cc:2726] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.284854 4746 leader_election.cc:341] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.285001 4746 leader_election.cc:302] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.285235 6349 raft_consensus.cc:2726] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.285452 4746 leader_election.cc:341] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.288998 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20242c84f64969865d9eda5920bd78 (DEFAULT_TABLE table=default.Table_656a2eccad1e4e59aa586fc88c5d7f89 [id=bd11281bf8564aa4a34b33537d1b32d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.288988 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20242c84f64969865d9eda5920bd78 (DEFAULT_TABLE table=default.Table_656a2eccad1e4e59aa586fc88c5d7f89 [id=bd11281bf8564aa4a34b33537d1b32d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.289413 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20242c84f64969865d9eda5920bd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.289541 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20242c84f64969865d9eda5920bd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.295154 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7fca4edc91784e4aa3381423e42e2d8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:41.295686 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 770) took 1656 ms. Trace:
W20240503 02:12:41.295781 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fca4edc91784e4aa3381423e42e2d8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.296197 5159 rpcz_store.cc:276] 0503 02:12:39.639256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.644252 (+ 4996us) service_pool.cc:225] Handling call
0503 02:12:41.147834 (+1503582us) ts_tablet_manager.cc:1120] Deleting tablet 7461aa76a1af41cb9253285bda708f30
0503 02:12:41.147844 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.167864 (+ 20020us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.192374 (+ 24510us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.295663 (+103289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9644,"tablet-delete.queue_time_us":1503445}
I20240503 02:12:41.296798 5159 tablet_replica.cc:330] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:41.297070 5159 raft_consensus.cc:2229] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.297132 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_de11f8796c984bd79ffadd090787b9a5" } new_table_name: "default.Table_6c8a25a0ec7945e39e6f222587404269" modify_external_catalogs: true
I20240503 02:12:41.297307 5159 raft_consensus.cc:2258] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.298588 5103 tablet_bootstrap.cc:492] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.298722 5103 ts_tablet_manager.cc:1367] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.000s
I20240503 02:12:41.298724 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.299252 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.299546 5103 raft_consensus.cc:348] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.299844 5103 raft_consensus.cc:374] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.299912 5103 raft_consensus.cc:729] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.300211 5103 consensus_queue.cc:260] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.300405 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6a0d5b09dfb04a1892fb248b676a923b" } new_table_name: "default.Table_0d5505a598db4432af3d4cbe09a8aa3d" modify_external_catalogs: true
I20240503 02:12:41.302735 5105 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.302868 5105 ts_tablet_manager.cc:1367] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.157s user 0.004s sys 0.002s
I20240503 02:12:41.305155 5105 raft_consensus.cc:348] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.305459 5105 raft_consensus.cc:374] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.305559 5105 raft_consensus.cc:729] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.305526 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0b12b36e91cd4d519c2e1277d3d97637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.306066 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b12b36e91cd4d519c2e1277d3d97637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.305923 5105 consensus_queue.cc:260] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.306666 5105 ts_tablet_manager.cc:1398] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:41.306989 5105 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.307379 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1829205c0cf64fd3ab99c62e91ab9229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.307868 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1829205c0cf64fd3ab99c62e91ab9229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.308997 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_75a0bf29850647249a6ed3ad0ca8885e" } new_table_name: "default.Table_6c31c5686db9455c836948d8e8abbfe8" modify_external_catalogs: true
I20240503 02:12:41.310858 5105 tablet_bootstrap.cc:654] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.310899 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_acdeb2544a5243f184185b6c97b607f8" } new_table_name: "default.Table_75da2b9d49e2436e96e15d6af2508d1a" modify_external_catalogs: true
I20240503 02:12:41.312104 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_82219dec0a7945719fe7415d05e92a13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.312605 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82219dec0a7945719fe7415d05e92a13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:41.289155 4746 leader_election.cc:341] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.313930 4746 leader_election.cc:302] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.316633 6349 raft_consensus.cc:2726] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:41.317798 4462 leader_election.cc:341] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 7178760c21904207b6c8317093871f63
I20240503 02:12:41.317924 4462 leader_election.cc:302] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:41.318130 6560 raft_consensus.cc:2726] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.319281 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_56cb6b9e2faa4ecb8a243d55becff404"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.319882 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56cb6b9e2faa4ecb8a243d55becff404 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.322463 6374 raft_consensus.cc:504] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.323043 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 700) took 3179 ms. Trace:
I20240503 02:12:41.323902 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c48b14bc624c28a871d45e9429ee30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.324290 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8656a886564f06931039b516162b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.325019 6374 leader_election.cc:288] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.325636 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 9003cc376aa149b59947bc32bd071163 (DEFAULT_TABLE table=default.Table_4b3d4e5854b34468a48715e9093f70e2 [id=5c675fea25ce4e6b87395b51899da369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.325640 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 7279550e731e440d92bf343c60a9497b (DEFAULT_TABLE table=default.Table_cea7d864ad774bb5b4b975e8e72989ad [id=ed2b2721928f489580ed5f1745060b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.326014 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7279550e731e440d92bf343c60a9497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.326104 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 7279550e731e440d92bf343c60a9497b (DEFAULT_TABLE table=default.Table_cea7d864ad774bb5b4b975e8e72989ad [id=ed2b2721928f489580ed5f1745060b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.326509 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7279550e731e440d92bf343c60a9497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.326902 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 4e848d895d6940998c33727b4c9c6136 (DEFAULT_TABLE table=default.Table_cfbd41260f214e91ab9c1fec294cef84 [id=cc51266fdfbb4f4a9897d977ed905910]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.327160 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]) successfully deleted
I20240503 02:12:41.327664 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e848d895d6940998c33727b4c9c6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.327999 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7178760c21904207b6c8317093871f63 (table default.Table_7d5419d4bbf34b67bcc05cf32663c2df [id=07e16e4056424605a70ccfaa25a24613]) successfully deleted
W20240503 02:12:41.328263 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5cd296bb3b574d6cbb0252ad10ada00c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cd296bb3b574d6cbb0252ad10ada00c already in progress: creating tablet
I20240503 02:12:41.329324 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 587d9e985b9b42bf8b3b2422f48a983a (DEFAULT_TABLE table=default.Table_aebba9d5877c44288da2f44d9e3b2e33 [id=44c87c2dc5c743928c2a3cd0fa76d96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.329667 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587d9e985b9b42bf8b3b2422f48a983a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.333245 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 587d9e985b9b42bf8b3b2422f48a983a (DEFAULT_TABLE table=default.Table_aebba9d5877c44288da2f44d9e3b2e33 [id=44c87c2dc5c743928c2a3cd0fa76d96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.333617 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587d9e985b9b42bf8b3b2422f48a983a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.334807 5159 ts_tablet_manager.cc:1875] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.312883 5114 ts_tablet_manager.cc:1888] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.346133 5114 log.cc:1196] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:41.346644 5114 ts_tablet_manager.cc:1909] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.347537 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 704) took 3072 ms. Trace:
I20240503 02:12:41.347723 5114 rpcz_store.cc:276] 0503 02:12:38.274854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.276633 (+ 1779us) service_pool.cc:225] Handling call
0503 02:12:41.227299 (+2950666us) ts_tablet_manager.cc:1120] Deleting tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
0503 02:12:41.227312 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.289775 (+ 62463us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.312773 (+ 22998us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.347515 (+ 34742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950478}
I20240503 02:12:41.348371 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 702) took 3073 ms. Trace:
I20240503 02:12:41.348521 5114 rpcz_store.cc:276] 0503 02:12:38.274612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.277712 (+ 3100us) service_pool.cc:225] Handling call
0503 02:12:41.348271 (+3070559us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:41.348281 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.348357 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3070338}
I20240503 02:12:41.349011 5114 tablet_replica.cc:330] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.349329 5114 raft_consensus.cc:2229] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.349303 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c421694ce1e424da7953d061a6bca3e with delete_type TABLET_DATA_DELETED (replaced by tablet 03fc31800276415fa68d95339025a55f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.349588 5114 raft_consensus.cc:2258] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.350260 5114 ts_tablet_manager.cc:1875] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.355096 5103 ts_tablet_manager.cc:1398] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.056s user 0.001s sys 0.002s
I20240503 02:12:41.355684 5103 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.357914 5103 tablet_bootstrap.cc:654] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.361498 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 128f5818e0d24ae59e1af4e3f97595bc (DEFAULT_TABLE table=default.Table_949c4a7045e749d1839f5c9e29997ebd [id=6323e7f056964c1b82333f857f32bc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.361588 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88d274b7654ba4942e757ba3302101 (DEFAULT_TABLE table=default.Table_055c133ec2d14ff7b5940ff3d79ad420 [id=a25ebae9618947cba49f1f22e6b1e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.361953 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128f5818e0d24ae59e1af4e3f97595bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.362233 4781 tablet_service.cc:1467] Processing CreateTablet for tablet d24537f1de7243ad94f319d7a65873ac (DEFAULT_TABLE table=default.Table_f6d3358d9b15463ca0f4ac45092a4616 [id=c1330713510b4ce792be0153d1492466]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.362386 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 3f74025b47674c88aa2969b5ace4f691 (DEFAULT_TABLE table=default.Table_9d3caa7babae4afbaa0e5351cf84e421 [id=70f8e0c7aa224e67bf2105aa29465298]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.362640 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24537f1de7243ad94f319d7a65873ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.362835 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 01e3a4d6e43c4798af7f480b5930e368 (DEFAULT_TABLE table=default.Table_84278e0d870b4961a9ca3574619e6789 [id=33ea1430ab1e4963a10c0107f45139a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.363152 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e3a4d6e43c4798af7f480b5930e368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.365039 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f74025b47674c88aa2969b5ace4f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.365214 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 128f5818e0d24ae59e1af4e3f97595bc (DEFAULT_TABLE table=default.Table_949c4a7045e749d1839f5c9e29997ebd [id=6323e7f056964c1b82333f857f32bc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.361963 4507 tablet_service.cc:1467] Processing CreateTablet for tablet b616953fd20f4e8e8f9ee8a2fd81a07a (DEFAULT_TABLE table=default.Table_7abfa4c4ae61433f9ac98e07d9954442 [id=4016360c2b1749d68c8194b50ce66812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.365533 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128f5818e0d24ae59e1af4e3f97595bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.365828 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b616953fd20f4e8e8f9ee8a2fd81a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.368408 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88d274b7654ba4942e757ba3302101 (DEFAULT_TABLE table=default.Table_055c133ec2d14ff7b5940ff3d79ad420 [id=a25ebae9618947cba49f1f22e6b1e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.368731 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88d274b7654ba4942e757ba3302101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.369609 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88d274b7654ba4942e757ba3302101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.372020 4778 tablet_service.cc:1467] Processing CreateTablet for tablet b616953fd20f4e8e8f9ee8a2fd81a07a (DEFAULT_TABLE table=default.Table_7abfa4c4ae61433f9ac98e07d9954442 [id=4016360c2b1749d68c8194b50ce66812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.372350 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b616953fd20f4e8e8f9ee8a2fd81a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.372807 5105 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.372808 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.373035 5105 ts_tablet_manager.cc:1367] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20240503 02:12:41.374011 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 01e3a4d6e43c4798af7f480b5930e368 (DEFAULT_TABLE table=default.Table_84278e0d870b4961a9ca3574619e6789 [id=33ea1430ab1e4963a10c0107f45139a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.374107 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 3f74025b47674c88aa2969b5ace4f691 (DEFAULT_TABLE table=default.Table_9d3caa7babae4afbaa0e5351cf84e421 [id=70f8e0c7aa224e67bf2105aa29465298]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.374385 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e3a4d6e43c4798af7f480b5930e368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.374612 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f74025b47674c88aa2969b5ace4f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.374855 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 147fb22c2f1145fc98d80a32fde5a844 (DEFAULT_TABLE table=default.Table_1aa10835245349bcade6a83360c1983d [id=68c89a9f17d647aa98ab8aa87ea485fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.374837 5105 raft_consensus.cc:348] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.375165 5105 raft_consensus.cc:374] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.375214 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147fb22c2f1145fc98d80a32fde5a844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.375504 5105 raft_consensus.cc:729] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.376013 5105 consensus_queue.cc:260] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.381892 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 7f11d140ac414c0c97af4f31b6e407e5 (DEFAULT_TABLE table=default.Table_3c1ab72c974d423a9a3a53f7cf411fc1 [id=d87348e6509e401cbf8d0937dde7a061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.382534 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f11d140ac414c0c97af4f31b6e407e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.384029 4497 tablet_service.cc:1467] Processing CreateTablet for tablet d24537f1de7243ad94f319d7a65873ac (DEFAULT_TABLE table=default.Table_f6d3358d9b15463ca0f4ac45092a4616 [id=c1330713510b4ce792be0153d1492466]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.384405 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24537f1de7243ad94f319d7a65873ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.396595 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 7f11d140ac414c0c97af4f31b6e407e5 (DEFAULT_TABLE table=default.Table_3c1ab72c974d423a9a3a53f7cf411fc1 [id=d87348e6509e401cbf8d0937dde7a061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.323195 5115 rpcz_store.cc:276] 0503 02:12:38.144046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.145718 (+ 1672us) service_pool.cc:225] Handling call
0503 02:12:41.282537 (+3136819us) ts_tablet_manager.cc:1120] Deleting tablet 0e89c1cf6ed54d2fafb2173ea9373ab5
0503 02:12:41.282545 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.323023 (+ 40478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3136252}
I20240503 02:12:41.402750 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
W20240503 02:12:41.404517 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because the tablet was not found. No further retry: Not found: Tablet not found: 22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:41.404886 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 067eaec72a314d7a820ffe0adbcb75f8 (DEFAULT_TABLE table=default.Table_ecb8bc8e313042c292129fa02342ed2f [id=341076d5e2eb4c70b1d242be429eff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.393909 5105 ts_tablet_manager.cc:1398] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240503 02:12:41.406215 5105 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.408164 4638 tablet_service.cc:1467] Processing CreateTablet for tablet ecb2b30119ec43cdabdb2319222569c3 (DEFAULT_TABLE table=default.Table_348455618e27462284c9005cc2368dd7 [id=40fd861bb4f3406689ebe14e7ff2e9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.410449 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce96306714144bfb5ee05bbf4d25572 (DEFAULT_TABLE table=default.Table_1a6867e7591e44dc942982feb523e922 [id=4712f930b6984081956e2d9864f8c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.411177 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20242c84f64969865d9eda5920bd78 (DEFAULT_TABLE table=default.Table_656a2eccad1e4e59aa586fc88c5d7f89 [id=bd11281bf8564aa4a34b33537d1b32d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.411414 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 4e848d895d6940998c33727b4c9c6136 (DEFAULT_TABLE table=default.Table_cfbd41260f214e91ab9c1fec294cef84 [id=cc51266fdfbb4f4a9897d977ed905910]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.411582 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 7279550e731e440d92bf343c60a9497b (DEFAULT_TABLE table=default.Table_cea7d864ad774bb5b4b975e8e72989ad [id=ed2b2721928f489580ed5f1745060b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.411746 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 7f11d140ac414c0c97af4f31b6e407e5 (DEFAULT_TABLE table=default.Table_3c1ab72c974d423a9a3a53f7cf411fc1 [id=d87348e6509e401cbf8d0937dde7a061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.411911 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 587d9e985b9b42bf8b3b2422f48a983a (DEFAULT_TABLE table=default.Table_aebba9d5877c44288da2f44d9e3b2e33 [id=44c87c2dc5c743928c2a3cd0fa76d96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.413085 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f11d140ac414c0c97af4f31b6e407e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.413206 5105 tablet_bootstrap.cc:654] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.415184 4639 tablet_service.cc:1467] Processing CreateTablet for tablet d24537f1de7243ad94f319d7a65873ac (DEFAULT_TABLE table=default.Table_f6d3358d9b15463ca0f4ac45092a4616 [id=c1330713510b4ce792be0153d1492466]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.415815 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 128f5818e0d24ae59e1af4e3f97595bc (DEFAULT_TABLE table=default.Table_949c4a7045e749d1839f5c9e29997ebd [id=6323e7f056964c1b82333f857f32bc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.417695 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 7f88d274b7654ba4942e757ba3302101 (DEFAULT_TABLE table=default.Table_055c133ec2d14ff7b5940ff3d79ad420 [id=a25ebae9618947cba49f1f22e6b1e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.418238 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 4e848d895d6940998c33727b4c9c6136 (DEFAULT_TABLE table=default.Table_cfbd41260f214e91ab9c1fec294cef84 [id=cc51266fdfbb4f4a9897d977ed905910]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.418855 4646 tablet_service.cc:1467] Processing CreateTablet for tablet b616953fd20f4e8e8f9ee8a2fd81a07a (DEFAULT_TABLE table=default.Table_7abfa4c4ae61433f9ac98e07d9954442 [id=4016360c2b1749d68c8194b50ce66812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.420060 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 3f74025b47674c88aa2969b5ace4f691 (DEFAULT_TABLE table=default.Table_9d3caa7babae4afbaa0e5351cf84e421 [id=70f8e0c7aa224e67bf2105aa29465298]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.420207 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 01e3a4d6e43c4798af7f480b5930e368 (DEFAULT_TABLE table=default.Table_84278e0d870b4961a9ca3574619e6789 [id=33ea1430ab1e4963a10c0107f45139a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.422685 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9003cc376aa149b59947bc32bd071163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.426331 4499 tablet_service.cc:1467] Processing CreateTablet for tablet db6fa251094842b9a629fdb67e20690b (DEFAULT_TABLE table=default.Table_8e5698a54a9249b2a78432ae0801f067 [id=fdb7a84657694e4db221af41719b767d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.426764 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6fa251094842b9a629fdb67e20690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.430752 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3f3967a2604dc2aa6a0f6f867fc298 (DEFAULT_TABLE table=default.Table_cf90801bade4413b98880d6506b87076 [id=7045776a018d49d797ce31695eb95558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.431128 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3f3967a2604dc2aa6a0f6f867fc298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.440090 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef1ef7398945e2b540eaf27163ca22 (DEFAULT_TABLE table=default.Table_0d7ac590920b4d4fb4fbc65cf513ef74 [id=00bc3e651c8f4ba1a7bd47f540310e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.440539 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef1ef7398945e2b540eaf27163ca22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.443321 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e848d895d6940998c33727b4c9c6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.447960 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 147fb22c2f1145fc98d80a32fde5a844 (DEFAULT_TABLE table=default.Table_1aa10835245349bcade6a83360c1983d [id=68c89a9f17d647aa98ab8aa87ea485fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.429044 4776 tablet_service.cc:1467] Processing CreateTablet for tablet db6fa251094842b9a629fdb67e20690b (DEFAULT_TABLE table=default.Table_8e5698a54a9249b2a78432ae0801f067 [id=fdb7a84657694e4db221af41719b767d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.438544 5105 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.441234 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3f3967a2604dc2aa6a0f6f867fc298 (DEFAULT_TABLE table=default.Table_cf90801bade4413b98880d6506b87076 [id=7045776a018d49d797ce31695eb95558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.449007 5105 ts_tablet_manager.cc:1367] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240503 02:12:41.418099 5159 ts_tablet_manager.cc:1888] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.441573 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 147fb22c2f1145fc98d80a32fde5a844 (DEFAULT_TABLE table=default.Table_1aa10835245349bcade6a83360c1983d [id=68c89a9f17d647aa98ab8aa87ea485fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.449402 5159 log.cc:1196] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/95681e67db1e4485876d530133e8a514
I20240503 02:12:41.449831 5159 ts_tablet_manager.cc:1909] T 95681e67db1e4485876d530133e8a514 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:41.451093 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 780) took 1805 ms. Trace:
I20240503 02:12:41.451308 5159 rpcz_store.cc:276] 0503 02:12:39.645382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.645666 (+ 284us) service_pool.cc:225] Handling call
0503 02:12:41.296751 (+1651085us) ts_tablet_manager.cc:1120] Deleting tablet 95681e67db1e4485876d530133e8a514
0503 02:12:41.296761 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.370291 (+ 73530us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.416883 (+ 46592us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.451072 (+ 34189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650911}
I20240503 02:12:41.451535 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6fa251094842b9a629fdb67e20690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.451884 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147fb22c2f1145fc98d80a32fde5a844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.454355 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3f3967a2604dc2aa6a0f6f867fc298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.457211 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:41.457414 4529 raft_consensus.cc:3037] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.458518 5159 tablet_replica.cc:330] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:41.460058 4529 raft_consensus.cc:2454] T 556f8938f35f42bfbbfb27f829653d45 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.460744 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef1ef7398945e2b540eaf27163ca22 (DEFAULT_TABLE table=default.Table_0d7ac590920b4d4fb4fbc65cf513ef74 [id=00bc3e651c8f4ba1a7bd47f540310e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.461390 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef1ef7398945e2b540eaf27163ca22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.461705 6296 raft_consensus.cc:482] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.462131 6296 raft_consensus.cc:504] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.462705 6296 leader_election.cc:288] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.462962 5159 raft_consensus.cc:2229] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.465647 5159 raft_consensus.cc:2258] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.465679 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 26c505fdcb70418c85bb913a3be1d334 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.469738 4499 tablet_service.cc:1467] Processing CreateTablet for tablet e930fc9d89e1499ab1c19185aed3444a (DEFAULT_TABLE table=default.Table_c5f5ee35b528431cb4e7256693a743a0 [id=3dec0c764f07410d976b36dbd80ae230]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.469779 5105 raft_consensus.cc:348] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.469949 5105 raft_consensus.cc:374] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.470023 5106 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.470243 5106 ts_tablet_manager.cc:1367] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.234s user 0.009s sys 0.001s
I20240503 02:12:41.471015 5106 raft_consensus.cc:348] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.471275 5106 raft_consensus.cc:374] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.471365 5106 raft_consensus.cc:729] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.471637 5106 consensus_queue.cc:260] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.472293 5106 ts_tablet_manager.cc:1398] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240503 02:12:41.472692 6294 log.cc:924] Time spent T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20240503 02:12:41.473229 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c505fdcb70418c85bb913a3be1d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.477017 6561 consensus_queue.cc:1035] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.478482 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e930fc9d89e1499ab1c19185aed3444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.470492 5159 ts_tablet_manager.cc:1875] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:41.477389 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 53427us
I20240503 02:12:41.477967 4781 tablet_service.cc:1467] Processing CreateTablet for tablet e930fc9d89e1499ab1c19185aed3444a (DEFAULT_TABLE table=default.Table_c5f5ee35b528431cb4e7256693a743a0 [id=3dec0c764f07410d976b36dbd80ae230]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.480171 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.480579 4507 tablet_service.cc:1467] Processing CreateTablet for tablet ea11546176f94074a18b91d528290e72 (DEFAULT_TABLE table=default.Table_d286219eeee948669a9681bd7e501192 [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.480960 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea11546176f94074a18b91d528290e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.481482 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.482170 4788 tablet_service.cc:1467] Processing CreateTablet for tablet ea11546176f94074a18b91d528290e72 (DEFAULT_TABLE table=default.Table_d286219eeee948669a9681bd7e501192 [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.479823 5105 raft_consensus.cc:729] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.483155 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 2da27c48da204cb384a99b0032029b63 (DEFAULT_TABLE table=default.Table_f973292ca29f4783af48772d007e24de [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.483628 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 2da27c48da204cb384a99b0032029b63 (DEFAULT_TABLE table=default.Table_f973292ca29f4783af48772d007e24de [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.485373 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 26c505fdcb70418c85bb913a3be1d334 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.496786 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067eaec72a314d7a820ffe0adbcb75f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.500098 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb2b30119ec43cdabdb2319222569c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.501230 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce96306714144bfb5ee05bbf4d25572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.501466 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7279550e731e440d92bf343c60a9497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.501642 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f11d140ac414c0c97af4f31b6e407e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.501912 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587d9e985b9b42bf8b3b2422f48a983a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.502132 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e848d895d6940998c33727b4c9c6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.505163 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24537f1de7243ad94f319d7a65873ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.505357 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128f5818e0d24ae59e1af4e3f97595bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.505743 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f88d274b7654ba4942e757ba3302101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.509209 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b616953fd20f4e8e8f9ee8a2fd81a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.509627 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f74025b47674c88aa2969b5ace4f691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.509948 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147fb22c2f1145fc98d80a32fde5a844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.513101 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e3a4d6e43c4798af7f480b5930e368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.517235 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20242c84f64969865d9eda5920bd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.519672 6296 raft_consensus.cc:482] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.519810 6296 raft_consensus.cc:504] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.521034 6262 raft_consensus.cc:482] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.521159 6262 raft_consensus.cc:504] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.521715 6262 leader_election.cc:288] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.522020 5106 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:41.524178 5106 tablet_bootstrap.cc:654] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.529827 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da27c48da204cb384a99b0032029b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.533780 5114 ts_tablet_manager.cc:1888] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.533972 5114 log.cc:1196] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/29e5c40f1f6b40308eed26b64d047494
I20240503 02:12:41.534616 5114 ts_tablet_manager.cc:1909] T 29e5c40f1f6b40308eed26b64d047494 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.535887 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 707) took 3252 ms. Trace:
I20240503 02:12:41.536065 5114 rpcz_store.cc:276] 0503 02:12:38.283932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.284774 (+ 842us) service_pool.cc:225] Handling call
0503 02:12:41.348961 (+3064187us) ts_tablet_manager.cc:1120] Deleting tablet 29e5c40f1f6b40308eed26b64d047494
0503 02:12:41.348970 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.418544 (+ 69574us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.533662 (+115118us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.535866 (+ 2204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063928}
I20240503 02:12:41.542207 5114 tablet_replica.cc:330] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.545465 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 701) took 3380 ms. Trace:
I20240503 02:12:41.545607 5115 rpcz_store.cc:276] 0503 02:12:38.164742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.170722 (+ 5980us) service_pool.cc:225] Handling call
0503 02:12:41.401158 (+3230436us) ts_tablet_manager.cc:1120] Deleting tablet 8f8b6599206549dd8ddf449a52c61fce
0503 02:12:41.496501 (+ 95343us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.545441 (+ 48940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3204241}
I20240503 02:12:41.546015 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 702) took 3126 ms. Trace:
I20240503 02:12:41.546115 5115 rpcz_store.cc:276] 0503 02:12:38.420037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.420276 (+ 239us) service_pool.cc:225] Handling call
0503 02:12:41.545909 (+3125633us) ts_tablet_manager.cc:1120] Deleting tablet 56aa7303f7784d2289a8da9653d1e317
0503 02:12:41.545918 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.546000 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3125422}
I20240503 02:12:41.546450 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 703) took 3125 ms. Trace:
I20240503 02:12:41.546542 5115 rpcz_store.cc:276] 0503 02:12:38.420856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.420931 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:41.546347 (+3125416us) ts_tablet_manager.cc:1120] Deleting tablet 20b6a18be64049049e1a083bf4af51ba
0503 02:12:41.546355 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.546437 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3125247}
I20240503 02:12:41.546804 5115 tablet_replica.cc:330] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:41.547009 5115 raft_consensus.cc:2229] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.547206 5115 raft_consensus.cc:2258] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.547902 5115 ts_tablet_manager.cc:1875] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.549527 4651 tablet_service.cc:1467] Processing CreateTablet for tablet db6fa251094842b9a629fdb67e20690b (DEFAULT_TABLE table=default.Table_8e5698a54a9249b2a78432ae0801f067 [id=fdb7a84657694e4db221af41719b767d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.550202 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6742a0406a9945e2ba8eb04f293bb1c9" } new_table_name: "default.Table_6815736795654b079117f0285ae63c1c" modify_external_catalogs: true
I20240503 02:12:41.551121 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3f3967a2604dc2aa6a0f6f867fc298 (DEFAULT_TABLE table=default.Table_cf90801bade4413b98880d6506b87076 [id=7045776a018d49d797ce31695eb95558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.551784 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8de2edcca99e40cdaf5db0ea01f4a652"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.552402 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8de2edcca99e40cdaf5db0ea01f4a652 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.553411 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6fa251094842b9a629fdb67e20690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.554848 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_866c8697e32646dc860d3cbc3856e41e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.555409 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_866c8697e32646dc860d3cbc3856e41e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.557389 5114 raft_consensus.cc:2229] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.557575 5114 raft_consensus.cc:2258] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.558100 5114 ts_tablet_manager.cc:1875] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:41.558176 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 consensus state has no leader
W20240503 02:12:41.558353 6294 log.cc:924] Time spent T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240503 02:12:41.558775 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ebe4e5a3a5124836ab1d84342f373932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:41.558988 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3f3967a2604dc2aa6a0f6f867fc298. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:41.559371 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebe4e5a3a5124836ab1d84342f373932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.561447 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5171aa8dfe0c451c902f936b9587191d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.562024 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5171aa8dfe0c451c902f936b9587191d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:41.563846 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0db84f8ed09342ea8b1c3b00b2036c50 consensus state has no leader
I20240503 02:12:41.564254 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_95aad9f7fca14d9fb0d98d25071f9f08" } modify_external_catalogs: true
I20240503 02:12:41.564445 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_95aad9f7fca14d9fb0d98d25071f9f08" } modify_external_catalogs: true
I20240503 02:12:41.569891 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 26c505fdcb70418c85bb913a3be1d334 (DEFAULT_TABLE table=default.Table_de08435101204e569789eb4f8e2ed689 [id=e5030d5f64d24f2086efa23991bcdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.570493 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a567b7af96934be8836963c85cc81473" } modify_external_catalogs: true
I20240503 02:12:41.570724 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a567b7af96934be8836963c85cc81473" } modify_external_catalogs: true
I20240503 02:12:41.571273 4644 tablet_service.cc:1467] Processing CreateTablet for tablet e930fc9d89e1499ab1c19185aed3444a (DEFAULT_TABLE table=default.Table_c5f5ee35b528431cb4e7256693a743a0 [id=3dec0c764f07410d976b36dbd80ae230]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:41.571563 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfcc73e2eb57429aa61b52b42198c747 consensus state has no leader
I20240503 02:12:41.571909 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_186402d8f9e2484c89977667d29d1ff9" } modify_external_catalogs: true
I20240503 02:12:41.571929 4643 tablet_service.cc:1467] Processing CreateTablet for tablet ea11546176f94074a18b91d528290e72 (DEFAULT_TABLE table=default.Table_d286219eeee948669a9681bd7e501192 [id=73b33f37dc3e4630a5935314a7597445]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.572093 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_186402d8f9e2484c89977667d29d1ff9" } modify_external_catalogs: true
I20240503 02:12:41.573823 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c505fdcb70418c85bb913a3be1d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.574147 5103 tablet_bootstrap.cc:492] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.574465 5103 ts_tablet_manager.cc:1367] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.026s
W20240503 02:12:41.544843 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3b9341be62a4275ad6e4c7d7b745621 consensus state has no leader
I20240503 02:12:41.576732 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_56845fb712d74923ac2ad61b5a1f6e3e" } modify_external_catalogs: true
I20240503 02:12:41.577008 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_56845fb712d74923ac2ad61b5a1f6e3e" } modify_external_catalogs: true
I20240503 02:12:41.577154 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e930fc9d89e1499ab1c19185aed3444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.577342 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea11546176f94074a18b91d528290e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.577134 5103 raft_consensus.cc:348] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.577489 5103 raft_consensus.cc:374] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.577597 5103 raft_consensus.cc:729] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.577939 5103 consensus_queue.cc:260] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.579658 5103 ts_tablet_manager.cc:1398] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:41.580627 5103 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.585721 5103 tablet_bootstrap.cc:654] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.593981 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e66e7c44c2784f97bc80c180112e3e11" } modify_external_catalogs: true
I20240503 02:12:41.594214 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e66e7c44c2784f97bc80c180112e3e11" } modify_external_catalogs: true
I20240503 02:12:41.597414 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 2da27c48da204cb384a99b0032029b63 (DEFAULT_TABLE table=default.Table_f973292ca29f4783af48772d007e24de [id=ac708eeb099f4828ba9c8ef3f9877c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.600652 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da27c48da204cb384a99b0032029b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.603693 5103 tablet_bootstrap.cc:492] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.604012 5103 ts_tablet_manager.cc:1367] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20240503 02:12:41.605593 5103 raft_consensus.cc:348] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.606020 5103 raft_consensus.cc:374] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.606185 5103 raft_consensus.cc:729] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.606662 5103 consensus_queue.cc:260] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:41.607302 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 116593us
I20240503 02:12:41.608186 5103 ts_tablet_manager.cc:1398] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:41.608635 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 95681e67db1e4485876d530133e8a514 (table default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]) successfully deleted
I20240503 02:12:41.609345 5103 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.610352 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9090b11c8a4bf0b4b92523b9431b79" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.610551 4528 raft_consensus.cc:2454] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.614568 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef1ef7398945e2b540eaf27163ca22 (DEFAULT_TABLE table=default.Table_0d7ac590920b4d4fb4fbc65cf513ef74 [id=00bc3e651c8f4ba1a7bd47f540310e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.615269 5103 tablet_bootstrap.cc:654] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.616232 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a90ebcf29004d1a925b69fa20998dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.616979 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e930fc9d89e1499ab1c19185aed3444a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.617349 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef1ef7398945e2b540eaf27163ca22. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:41.621750 4746 leader_election.cc:341] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.626519 5105 consensus_queue.cc:260] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.628522 5105 ts_tablet_manager.cc:1398] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.179s user 0.002s sys 0.000s
I20240503 02:12:41.629354 5105 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.629835 6374 consensus_queue.cc:1035] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.632553 6262 consensus_queue.cc:1035] T b28f66fe53064a46a0b98b78aa77115c P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.633054 5103 tablet_bootstrap.cc:492] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.633234 4608 leader_election.cc:302] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:41.633431 5103 ts_tablet_manager.cc:1367] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.006s
I20240503 02:12:41.633463 6262 raft_consensus.cc:2781] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.633550 6262 raft_consensus.cc:686] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.633756 6262 consensus_queue.cc:237] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.634407 6296 leader_election.cc:288] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.634929 5103 raft_consensus.cc:348] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.635246 5103 raft_consensus.cc:374] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.635329 5103 raft_consensus.cc:729] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.618639 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f8938f35f42bfbbfb27f829653d45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:41.635829 4608 leader_election.cc:302] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:41.635943 4811 raft_consensus.cc:3037] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.636034 6561 raft_consensus.cc:2781] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.635805 5103 consensus_queue.cc:260] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.636088 6561 raft_consensus.cc:482] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.636139 6561 raft_consensus.cc:3037] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.637296 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c505fdcb70418c85bb913a3be1d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.637168 5103 ts_tablet_manager.cc:1398] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:41.638284 6296 raft_consensus.cc:482] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.638408 6296 raft_consensus.cc:504] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.639323 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.639348 5103 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.639986 6349 raft_consensus.cc:482] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.640100 6349 raft_consensus.cc:504] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.640770 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da27c48da204cb384a99b0032029b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.641762 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea11546176f94074a18b91d528290e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.643028 6349 leader_election.cc:288] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.643771 6296 leader_election.cc:288] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.643334 6561 raft_consensus.cc:504] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.645754 6561 leader_election.cc:288] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.647042 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9090b11c8a4bf0b4b92523b9431b79" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:41.647235 4528 raft_consensus.cc:3037] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.651599 4528 raft_consensus.cc:2454] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.652113 5103 tablet_bootstrap.cc:654] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.655023 4811 raft_consensus.cc:2454] T 556f8938f35f42bfbbfb27f829653d45 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.668108 5103 tablet_bootstrap.cc:492] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.668445 5103 ts_tablet_manager.cc:1367] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20240503 02:12:41.669726 5103 raft_consensus.cc:348] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.670095 5103 raft_consensus.cc:374] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.670243 5103 raft_consensus.cc:729] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.670652 5103 consensus_queue.cc:260] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.672211 5103 ts_tablet_manager.cc:1398] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:41.673431 5103 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.678778 5103 tablet_bootstrap.cc:654] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.680022 5114 ts_tablet_manager.cc:1888] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.681924 5106 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.682101 5106 ts_tablet_manager.cc:1367] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.006s
I20240503 02:12:41.682370 5114 log.cc:1196] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/da12dd28e03f462b905d1844e2ad3340
I20240503 02:12:41.683068 5114 ts_tablet_manager.cc:1909] T da12dd28e03f462b905d1844e2ad3340 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.683969 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 708) took 3171 ms. Trace:
I20240503 02:12:41.684087 5114 rpcz_store.cc:276] 0503 02:12:38.512999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.513052 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:41.536536 (+3023484us) ts_tablet_manager.cc:1120] Deleting tablet da12dd28e03f462b905d1844e2ad3340
0503 02:12:41.536545 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.564013 (+ 27468us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.679913 (+115900us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.683945 (+ 4032us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15,"tablet-delete.queue_time_us":3023239}
I20240503 02:12:41.684573 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 720) took 2539 ms. Trace:
I20240503 02:12:41.689019 5114 rpcz_store.cc:276] 0503 02:12:39.145426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.153027 (+ 7601us) service_pool.cc:225] Handling call
0503 02:12:41.684440 (+2531413us) ts_tablet_manager.cc:1120] Deleting tablet a4a16052d43d45aeb405c5de5dafa041
0503 02:12:41.684450 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.684552 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2530702}
I20240503 02:12:41.690431 5105 tablet_bootstrap.cc:654] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.690600 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 723) took 2533 ms. Trace:
I20240503 02:12:41.690893 5114 rpcz_store.cc:276] 0503 02:12:39.156776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.156836 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:41.690292 (+2533456us) ts_tablet_manager.cc:1120] Deleting tablet a3374e095846463a859bb642c2f42167
0503 02:12:41.690324 (+ 32us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.690563 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2533214}
I20240503 02:12:41.692198 5106 raft_consensus.cc:348] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.692478 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 722) took 2535 ms. Trace:
I20240503 02:12:41.692598 5114 rpcz_store.cc:276] 0503 02:12:39.156639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.178169 (+ 21530us) service_pool.cc:225] Handling call
0503 02:12:41.692354 (+2514185us) ts_tablet_manager.cc:1120] Deleting tablet 16b3d1e7ce83425db0bd56457d9f2544
0503 02:12:41.692362 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.692457 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2513944}
I20240503 02:12:41.693421 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 721) took 2536 ms. Trace:
I20240503 02:12:41.695242 5106 raft_consensus.cc:374] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.696817 5106 raft_consensus.cc:729] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.696964 5114 rpcz_store.cc:276] 0503 02:12:39.156526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.179525 (+ 22999us) service_pool.cc:225] Handling call
0503 02:12:41.693309 (+2513784us) ts_tablet_manager.cc:1120] Deleting tablet 81d6d3ca80c54c629a5de82b06a7aac4
0503 02:12:41.693320 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.693404 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2508235}
I20240503 02:12:41.697476 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 724) took 2482 ms. Trace:
I20240503 02:12:41.697573 5114 rpcz_store.cc:276] 0503 02:12:39.215060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.215111 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:41.697373 (+2482262us) ts_tablet_manager.cc:1120] Deleting tablet fc67bba49712445db6d784b6ed94dd7e
0503 02:12:41.697381 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.697457 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2482036}
I20240503 02:12:41.697952 5114 tablet_replica.cc:330] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.698169 5114 raft_consensus.cc:2229] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.698346 5114 raft_consensus.cc:2258] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:41.702263 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6d08e7d17f1644749a9a2e7175b3191a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d08e7d17f1644749a9a2e7175b3191a already in progress: creating tablet
W20240503 02:12:41.705677 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0a8c57ddf3d8428e8fa5ffb061dc054b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a8c57ddf3d8428e8fa5ffb061dc054b already in progress: creating tablet
I20240503 02:12:41.706897 6374 raft_consensus.cc:482] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.707063 6374 raft_consensus.cc:504] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.697216 5106 consensus_queue.cc:260] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.708746 5114 ts_tablet_manager.cc:1875] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.707706 6374 leader_election.cc:288] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.714337 5103 tablet_bootstrap.cc:492] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.714682 5103 ts_tablet_manager.cc:1367] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.000s
W20240503 02:12:41.715092 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d84412b28b4c41c2a3ce18702403dcd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d84412b28b4c41c2a3ce18702403dcd8 already in progress: creating tablet
I20240503 02:12:41.716288 5103 raft_consensus.cc:348] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:41.716789 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 6b15997a94b74ad3808d965acfb5d155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b15997a94b74ad3808d965acfb5d155 already in progress: creating tablet
I20240503 02:12:41.716744 5103 raft_consensus.cc:374] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.716879 5103 raft_consensus.cc:729] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.717690 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.717868 4528 raft_consensus.cc:2454] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.718116 5103 consensus_queue.cc:260] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:41.718395 4746 leader_election.cc:341] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.718544 4746 leader_election.cc:302] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.719652 6296 raft_consensus.cc:2726] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.720185 5103 ts_tablet_manager.cc:1398] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:41.721083 5106 ts_tablet_manager.cc:1398] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
W20240503 02:12:41.721242 4746 leader_election.cc:341] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.721766 5103 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.721913 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.723102 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.723177 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b149e66a0446eeade190dd7f5d2bec" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.723877 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b149e66a0446eeade190dd7f5d2bec" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.723922 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f39e8d17b44895bd5c979d82145ad9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.724589 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f39e8d17b44895bd5c979d82145ad9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.725147 5106 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:41.726851 4746 leader_election.cc:341] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.726994 4746 leader_election.cc:302] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.727239 6296 raft_consensus.cc:2726] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.727581 6560 raft_consensus.cc:482] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:41.727658 4746 leader_election.cc:341] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.727713 6560 raft_consensus.cc:504] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:41.727921 4746 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.728338 6560 leader_election.cc:288] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.728950 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:41.729701 4746 leader_election.cc:341] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.729831 4746 leader_election.cc:302] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.730060 6296 raft_consensus.cc:2726] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.730485 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:41.730693 4811 raft_consensus.cc:2454] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.732165 5103 tablet_bootstrap.cc:654] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.733340 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:41.733963 4608 consensus_peers.cc:485] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7461aa76a1af41cb9253285bda708f30. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.737686 5159 ts_tablet_manager.cc:1888] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.737869 5159 log.cc:1196] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/59855ab3e42b4e019b769007ccbbc360
I20240503 02:12:41.738253 5159 ts_tablet_manager.cc:1909] T 59855ab3e42b4e019b769007ccbbc360 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:41.738560 4608 leader_election.cc:341] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 6c9090b11c8a4bf0b4b92523b9431b79
W20240503 02:12:41.739111 4608 leader_election.cc:341] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 6c9090b11c8a4bf0b4b92523b9431b79
I20240503 02:12:41.739213 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 779) took 2094 ms. Trace:
I20240503 02:12:41.739351 5159 rpcz_store.cc:276] 0503 02:12:39.644900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.645866 (+ 966us) service_pool.cc:225] Handling call
0503 02:12:41.458410 (+1812544us) ts_tablet_manager.cc:1120] Deleting tablet 59855ab3e42b4e019b769007ccbbc360
0503 02:12:41.458446 (+ 36us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.682990 (+224544us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.737565 (+ 54575us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.739187 (+ 1622us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":356,"tablet-delete.queue_time_us":1812397}
I20240503 02:12:41.739956 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 782) took 2088 ms. Trace:
W20240503 02:12:41.740648 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e89c1cf6ed54d2fafb2173ea9373ab5 already in progress: creating tablet
I20240503 02:12:41.741117 5114 ts_tablet_manager.cc:1888] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:41.742022 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8f8b6599206549dd8ddf449a52c61fce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8b6599206549dd8ddf449a52c61fce already in progress: creating tablet
W20240503 02:12:41.742476 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 56aa7303f7784d2289a8da9653d1e317 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56aa7303f7784d2289a8da9653d1e317 already in progress: creating tablet
W20240503 02:12:41.742736 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 20b6a18be64049049e1a083bf4af51ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b6a18be64049049e1a083bf4af51ba already in progress: creating tablet
W20240503 02:12:41.743291 4608 consensus_peers.cc:485] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.744077 4608 leader_election.cc:302] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:41.744387 6262 raft_consensus.cc:2781] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.744463 6262 raft_consensus.cc:482] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.744532 6262 raft_consensus.cc:3037] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.745077 5114 log.cc:1196] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/4225c1bc133d4e18a4a47fd58820ca10
I20240503 02:12:41.745498 5114 ts_tablet_manager.cc:1909] T 4225c1bc133d4e18a4a47fd58820ca10 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.746222 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 731) took 2356 ms. Trace:
I20240503 02:12:41.746348 5114 rpcz_store.cc:276] 0503 02:12:39.389693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.389743 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:41.697906 (+2308163us) ts_tablet_manager.cc:1120] Deleting tablet 4225c1bc133d4e18a4a47fd58820ca10
0503 02:12:41.697917 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.708564 (+ 10647us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x611000966470. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f13a67bef9d 00007f139fd3805e 00007f139fe65fe7 00007f13a28f957c 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74 00007f139336719b
0503 02:12:41.737542 (+ 28978us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.741021 (+ 3479us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.746203 (+ 5182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21793536,"tablet-delete.queue_time_us":2307989}
I20240503 02:12:41.746904 5114 tablet_replica.cc:330] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:41.751421 5114 raft_consensus.cc:2229] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.747289 5103 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.751690 5103 ts_tablet_manager.cc:1367] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.002s
I20240503 02:12:41.752614 5103 raft_consensus.cc:348] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.752847 5103 raft_consensus.cc:374] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.752964 5103 raft_consensus.cc:729] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.753219 5103 consensus_queue.cc:260] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.753808 5103 ts_tablet_manager.cc:1398] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:41.753484 5105 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.762274 5105 ts_tablet_manager.cc:1367] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.003s
I20240503 02:12:41.762167 5159 rpcz_store.cc:276] 0503 02:12:39.651934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.651980 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:41.739844 (+2087864us) ts_tablet_manager.cc:1120] Deleting tablet 7178760c21904207b6c8317093871f63
0503 02:12:41.739855 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.739937 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083264}
I20240503 02:12:41.763135 5105 raft_consensus.cc:348] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.763378 5105 raft_consensus.cc:374] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.763437 5105 raft_consensus.cc:729] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.763615 5105 consensus_queue.cc:260] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.764170 5105 ts_tablet_manager.cc:1398] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:41.764454 5105 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.764510 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 783) took 2112 ms. Trace:
I20240503 02:12:41.765237 5103 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.764889 5159 rpcz_store.cc:276] 0503 02:12:39.652190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.652239 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:41.763918 (+2111679us) ts_tablet_manager.cc:1120] Deleting tablet f2f9b962c9264f87b798df33544cb15d
0503 02:12:41.763929 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.764335 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2107086}
I20240503 02:12:41.766816 5103 tablet_bootstrap.cc:654] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.767323 5105 tablet_bootstrap.cc:654] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.767758 5114 raft_consensus.cc:2258] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:41.761637 4746 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.738098 5106 tablet_bootstrap.cc:654] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.767319 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 29e5c40f1f6b40308eed26b64d047494 (table default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]) successfully deleted
I20240503 02:12:41.768375 4746 leader_election.cc:302] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.768543 5114 ts_tablet_manager.cc:1875] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.768183 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 784) took 2115 ms. Trace:
I20240503 02:12:41.768762 6595 raft_consensus.cc:2726] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.769703 5159 rpcz_store.cc:276] 0503 02:12:39.652620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.652669 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:41.766761 (+2114092us) ts_tablet_manager.cc:1120] Deleting tablet a74f4816cad04a9eb5934aaba86c2fbc
0503 02:12:41.767101 (+ 340us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.767794 (+ 693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2110044}
I20240503 02:12:41.771790 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87713e899d2148308316d807a6d76921" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:41.728191 5115 ts_tablet_manager.cc:1888] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:41.772166 5115 log.cc:1196] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/da12dd28e03f462b905d1844e2ad3340
I20240503 02:12:41.772184 4811 raft_consensus.cc:2379] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:41.772630 5115 ts_tablet_manager.cc:1909] T da12dd28e03f462b905d1844e2ad3340 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:41.772903 4462 leader_election.cc:302] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:41.773038 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 704) took 3217 ms. Trace:
I20240503 02:12:41.773219 6560 raft_consensus.cc:3037] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.774768 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 785) took 2120 ms. Trace:
I20240503 02:12:41.774518 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet da12dd28e03f462b905d1844e2ad3340 (table default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]) successfully deleted
I20240503 02:12:41.775848 6560 raft_consensus.cc:2726] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.773119 5115 rpcz_store.cc:276] 0503 02:12:38.555157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.555882 (+ 725us) service_pool.cc:225] Handling call
0503 02:12:41.546766 (+2990884us) ts_tablet_manager.cc:1120] Deleting tablet da12dd28e03f462b905d1844e2ad3340
0503 02:12:41.546773 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.606152 (+ 59379us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.728068 (+121916us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.773016 (+ 44948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2990681}
I20240503 02:12:41.776860 5115 tablet_replica.cc:330] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:41.777144 5115 raft_consensus.cc:2229] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.777415 5115 raft_consensus.cc:2258] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.778021 5115 ts_tablet_manager.cc:1875] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.775868 5159 rpcz_store.cc:276] 0503 02:12:39.654344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.654436 (+ 92us) service_pool.cc:225] Handling call
0503 02:12:41.773667 (+2119231us) ts_tablet_manager.cc:1120] Deleting tablet 3b72c20a4b5a4d2ea266f031733d1aab
0503 02:12:41.773678 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.774395 (+ 717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116315}
I20240503 02:12:41.751008 6262 raft_consensus.cc:504] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.779063 6262 leader_election.cc:288] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.779464 4608 leader_election.cc:302] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:41.779996 6374 raft_consensus.cc:2781] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.780103 6374 raft_consensus.cc:686] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.780138 5114 ts_tablet_manager.cc:1888] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.780236 5114 log.cc:1196] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/6c9090b11c8a4bf0b4b92523b9431b79
W20240503 02:12:41.778445 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a4a16052d43d45aeb405c5de5dafa041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4a16052d43d45aeb405c5de5dafa041 already in progress: creating tablet
I20240503 02:12:41.780356 6374 consensus_queue.cc:237] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.780529 5114 ts_tablet_manager.cc:1909] T 6c9090b11c8a4bf0b4b92523b9431b79 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:41.780658 5159 tablet_replica.cc:330] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:41.782748 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a3374e095846463a859bb642c2f42167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3374e095846463a859bb642c2f42167 already in progress: creating tablet
W20240503 02:12:41.783430 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 81d6d3ca80c54c629a5de82b06a7aac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81d6d3ca80c54c629a5de82b06a7aac4 already in progress: creating tablet
I20240503 02:12:41.784296 5159 raft_consensus.cc:2229] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.786659 5159 raft_consensus.cc:2258] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.788210 5105 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.788384 5105 ts_tablet_manager.cc:1367] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20240503 02:12:41.789208 5105 raft_consensus.cc:348] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.789420 5105 raft_consensus.cc:374] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.789507 5105 raft_consensus.cc:729] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.789795 5103 tablet_bootstrap.cc:492] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:41.789767 5105 consensus_queue.cc:260] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.789989 5103 ts_tablet_manager.cc:1367] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240503 02:12:41.790448 5105 ts_tablet_manager.cc:1398] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:41.790716 5103 raft_consensus.cc:348] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.790957 5103 raft_consensus.cc:374] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.791043 5103 raft_consensus.cc:729] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.791311 5103 consensus_queue.cc:260] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.791833 5103 ts_tablet_manager.cc:1398] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:41.791868 5105 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.792326 5159 ts_tablet_manager.cc:1875] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.793722 5105 tablet_bootstrap.cc:654] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.802603 6560 consensus_queue.cc:1035] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:41.808528 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fc67bba49712445db6d784b6ed94dd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc67bba49712445db6d784b6ed94dd7e already in progress: creating tablet
I20240503 02:12:41.809305 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 4225c1bc133d4e18a4a47fd58820ca10 (table default.Table_bb63ba232a074c0692fa657db43e25dc [id=5cbfe61d82174908b3b00ff5d287eb8f]) successfully deleted
I20240503 02:12:41.821431 5103 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:41.822635 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet da12dd28e03f462b905d1844e2ad3340 (table default.Table_96ec56aa54e84e89a80c091d6bcd5e2f [id=8ac332b8aecc4c0d88efe869e1547b3f]) successfully deleted
W20240503 02:12:41.827145 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77fa5be5a4f64c189ef619bc38e21eae consensus state has no leader
I20240503 02:12:41.833354 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 7982ba6234b844f682cb23c0bfc4068c from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:41.833762 5106 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:41.833911 5106 ts_tablet_manager.cc:1367] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20240503 02:12:41.834650 5106 raft_consensus.cc:348] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.834846 5106 raft_consensus.cc:374] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.834916 5106 raft_consensus.cc:729] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.835111 5106 consensus_queue.cc:260] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.837162 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 70e08c10c15f41878202cef0c8d5ad3c
I20240503 02:12:41.838106 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f124b2c9e27d43678750918ac22d43c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.838742 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f124b2c9e27d43678750918ac22d43c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.839967 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:41.840694 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7b24aa2076944be5bc2a5fbf74726758"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.841231 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b24aa2076944be5bc2a5fbf74726758 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.841363 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet c88eaba979394b449d96ee956c3f28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.841986 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.843184 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7b86a6bb791449b5a4c490663d87afd4
I20240503 02:12:41.843958 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e0dab12ce8a54de18f9e00962d9a1c6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.844494 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0dab12ce8a54de18f9e00962d9a1c6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.845468 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_206d524a9f2542b9847dc17ed534ae92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.845919 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_206d524a9f2542b9847dc17ed534ae92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.847355 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.848268 5106 ts_tablet_manager.cc:1398] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:41.848600 5106 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:41.853250 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7982ba6234b844f682cb23c0bfc4068c
I20240503 02:12:41.854413 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 7982ba6234b844f682cb23c0bfc4068c deleted and replaced by tablet ef64cff211664efab3dc41a1f26c3de1
I20240503 02:12:41.855670 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7982ba6234b844f682cb23c0bfc4068c with delete_type TABLET_DATA_DELETED (replaced by tablet ef64cff211664efab3dc41a1f26c3de1) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.858899 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ee16f1beb6a34a85bc872dedd64472ce
I20240503 02:12:41.860162 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet ee16f1beb6a34a85bc872dedd64472ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:41.861173 5106 tablet_bootstrap.cc:654] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.863315 5159 ts_tablet_manager.cc:1888] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.864149 5159 log.cc:1196] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/87713e899d2148308316d807a6d76921
I20240503 02:12:41.861726 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2843778966d549a18acea39be8ad6929"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.866153 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2843778966d549a18acea39be8ad6929 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.868297 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_d286219eeee948669a9681bd7e501192" } new_table_name: "default.Table_155fde3f789341069f58c427a2c25eec" modify_external_catalogs: true
I20240503 02:12:41.870370 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7be6ff0537114d788c876ce810d2ae5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:41.871095 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7be6ff0537114d788c876ce810d2ae5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:41.872689 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9988e5c156a84cc4a4fd9b745f10867a" } new_table_name: "default.Table_eb85e71e0fe24211a3cdf53b81d20fba" modify_external_catalogs: true
I20240503 02:12:41.869714 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7fca4edc91784e4aa3381423e42e2d8d" } new_table_name: "default.Table_cc721583b82d4bc6b7a31189e77a3c17" modify_external_catalogs: true
I20240503 02:12:41.875312 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4e1bf750865941a9b815ad9f398cda31" } new_table_name: "default.Table_4d9a0ac169f94dfbad1496820ded5f07" modify_external_catalogs: true
I20240503 02:12:41.876458 6374 consensus_queue.cc:1035] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.877396 6595 raft_consensus.cc:482] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.877540 6595 raft_consensus.cc:504] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.878108 6595 leader_election.cc:288] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:41.822322 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:41.878316 4529 raft_consensus.cc:3037] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.879371 5103 tablet_bootstrap.cc:654] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.880549 6374 raft_consensus.cc:482] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.880721 6374 raft_consensus.cc:504] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.838256 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5e4db5204e4acf8719100d02d533fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.838389 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet b9511e4de6c942b9b9e0790c88d3d543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.838474 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 87713e899d2148308316d807a6d76921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.838832 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.841372 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet c88eaba979394b449d96ee956c3f28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.844274 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.854147 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7982ba6234b844f682cb23c0bfc4068c with delete_type TABLET_DATA_DELETED (replaced by tablet ef64cff211664efab3dc41a1f26c3de1) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.860141 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet ee16f1beb6a34a85bc872dedd64472ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:41.881942 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.882145 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.882103 5159 ts_tablet_manager.cc:1909] T 87713e899d2148308316d807a6d76921 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:41.882555 4746 leader_election.cc:341] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:41.882786 4746 leader_election.cc:341] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.882869 4746 leader_election.cc:302] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.883162 6374 leader_election.cc:288] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.883312 6595 raft_consensus.cc:2726] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.883590 4529 raft_consensus.cc:2454] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.884069 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.884346 4529 raft_consensus.cc:2454] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.885006 4608 leader_election.cc:302] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:41.885466 4608 leader_election.cc:302] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:41.885794 6317 raft_consensus.cc:2781] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.885924 6317 raft_consensus.cc:686] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.886236 6317 consensus_queue.cc:237] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.887490 6579 raft_consensus.cc:2781] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.887199 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 786) took 2232 ms. Trace:
I20240503 02:12:41.887584 6579 raft_consensus.cc:482] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.887661 6579 raft_consensus.cc:3037] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.887634 5159 rpcz_store.cc:276] 0503 02:12:39.654644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.654715 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:41.780247 (+2125532us) ts_tablet_manager.cc:1120] Deleting tablet 87713e899d2148308316d807a6d76921
0503 02:12:41.780265 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.834078 (+ 53813us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.862743 (+ 28665us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.887014 (+ 24271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123020}
W20240503 02:12:41.889434 4259 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 77fa5be5a4f64c189ef619bc38e21eae (table default.Table_6815736795654b079117f0285ae63c1c [id=a95b6d3cb7fd4a86931abd20d16664fc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 77fa5be5a4f64c189ef619bc38e21eae consensus state has no leader
I20240503 02:12:41.890309 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 781) took 2238 ms. Trace:
I20240503 02:12:41.890795 5105 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.890908 5105 ts_tablet_manager.cc:1367] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.003s
I20240503 02:12:41.890794 6579 raft_consensus.cc:504] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.890703 5159 rpcz_store.cc:276] 0503 02:12:39.651614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.652130 (+ 516us) service_pool.cc:225] Handling call
0503 02:12:41.889564 (+2237434us) ts_tablet_manager.cc:1120] Deleting tablet 7394c9d374404fb199ab6ff170d7febe
0503 02:12:41.889729 (+ 165us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.890133 (+ 404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232735}
I20240503 02:12:41.891477 6579 leader_election.cc:288] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:41.891690 5105 raft_consensus.cc:348] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.891913 5105 raft_consensus.cc:374] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.891990 5105 raft_consensus.cc:729] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.892258 5105 consensus_queue.cc:260] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.893028 5105 ts_tablet_manager.cc:1398] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:41.892870 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 787) took 2237 ms. Trace:
I20240503 02:12:41.893257 5159 rpcz_store.cc:276] 0503 02:12:39.655094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.655165 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:41.892120 (+2236955us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:41.892291 (+ 171us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.892697 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":2235283}
I20240503 02:12:41.893769 5105 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.896467 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 788) took 2240 ms. Trace:
I20240503 02:12:41.896692 5159 rpcz_store.cc:276] 0503 02:12:39.655587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.655643 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:41.893919 (+2238276us) ts_tablet_manager.cc:1120] Deleting tablet 885129a4efc5475b97b7ac68e1db5e1d
0503 02:12:41.896358 (+ 2439us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.896445 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2237032}
I20240503 02:12:41.897161 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 789) took 2238 ms. Trace:
I20240503 02:12:41.893851 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 59855ab3e42b4e019b769007ccbbc360 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
I20240503 02:12:41.897591 5159 rpcz_store.cc:276] 0503 02:12:39.658363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.658414 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:41.897071 (+2238657us) ts_tablet_manager.cc:1120] Deleting tablet 2152f205cf1b4ec7ae03605bbfa007e1
0503 02:12:41.897082 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.897144 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2238511}
W20240503 02:12:41.898023 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7178760c21904207b6c8317093871f63 because the tablet was not found. No further retry: Not found: Tablet not found: 7178760c21904207b6c8317093871f63
W20240503 02:12:41.898293 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f2f9b962c9264f87b798df33544cb15d because the tablet was not found. No further retry: Not found: Tablet not found: f2f9b962c9264f87b798df33544cb15d
I20240503 02:12:41.896627 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
W20240503 02:12:41.898522 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a74f4816cad04a9eb5934aaba86c2fbc because the tablet was not found. No further retry: Not found: Tablet not found: a74f4816cad04a9eb5934aaba86c2fbc
W20240503 02:12:41.898747 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3b72c20a4b5a4d2ea266f031733d1aab because the tablet was not found. No further retry: Not found: Tablet not found: 3b72c20a4b5a4d2ea266f031733d1aab
I20240503 02:12:41.898916 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 87713e899d2148308316d807a6d76921 (table default.Table_4702ebe6617c4aebb1264298076c6a4d [id=31669972ad634736b38958103d7b6c16]) successfully deleted
W20240503 02:12:41.899149 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7394c9d374404fb199ab6ff170d7febe because the tablet was not found. No further retry: Not found: Tablet not found: 7394c9d374404fb199ab6ff170d7febe
I20240503 02:12:41.899504 6595 raft_consensus.cc:482] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.902171 6595 raft_consensus.cc:504] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.902835 6595 leader_election.cc:288] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.902925 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e08c10c15f41878202cef0c8d5ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.900776 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.901722 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.904032 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.904186 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet c88eaba979394b449d96ee956c3f28a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.904489 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.898347 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 790) took 2239 ms. Trace:
I20240503 02:12:41.904660 5159 rpcz_store.cc:276] 0503 02:12:39.658791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.658845 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:41.898265 (+2239420us) ts_tablet_manager.cc:1120] Deleting tablet 5f56701ed6424c60bf1a985eea1936fe
0503 02:12:41.898275 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.898332 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239289}
W20240503 02:12:41.905006 4746 leader_election.cc:341] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.905041 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 791) took 2245 ms. Trace:
I20240503 02:12:41.905117 5159 rpcz_store.cc:276] 0503 02:12:39.659206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.659256 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:41.904949 (+2245693us) ts_tablet_manager.cc:1120] Deleting tablet 29e5c40f1f6b40308eed26b64d047494
0503 02:12:41.904960 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.905024 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245492}
I20240503 02:12:41.905371 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86a6bb791449b5a4c490663d87afd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:41.905834 4746 leader_election.cc:341] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.905962 4746 leader_election.cc:302] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:41.906001 4462 consensus_peers.cc:485] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d39f042afb954daa923502ef12377bd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.906360 6595 raft_consensus.cc:2726] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.906801 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 7982ba6234b844f682cb23c0bfc4068c with delete_type TABLET_DATA_DELETED (replaced by tablet ef64cff211664efab3dc41a1f26c3de1) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.907678 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet ee16f1beb6a34a85bc872dedd64472ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:41.908601 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a90ebcf29004d1a925b69fa20998dd3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:41.908803 4811 raft_consensus.cc:3037] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.905825 5105 tablet_bootstrap.cc:654] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.905416 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 792) took 2245 ms. Trace:
I20240503 02:12:41.909688 5159 rpcz_store.cc:276] 0503 02:12:39.659724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.659778 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:41.905341 (+2245563us) ts_tablet_manager.cc:1120] Deleting tablet 41ef3c6e5e7d40249b112144975ba220
0503 02:12:41.905349 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.905403 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245395}
I20240503 02:12:41.910061 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 793) took 2209 ms. Trace:
I20240503 02:12:41.910138 5159 rpcz_store.cc:276] 0503 02:12:39.700453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.706112 (+ 5659us) service_pool.cc:225] Handling call
0503 02:12:41.909939 (+2203827us) ts_tablet_manager.cc:1120] Deleting tablet 9fa3df92b6e544de9ca76d98706eebec
0503 02:12:41.909950 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.910047 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203630}
I20240503 02:12:41.910455 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 796) took 2026 ms. Trace:
I20240503 02:12:41.910519 5159 rpcz_store.cc:276] 0503 02:12:39.883914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.883975 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:41.910358 (+2026383us) ts_tablet_manager.cc:1120] Deleting tablet ab093285c39b4ce7a0bb3de9d016a918
0503 02:12:41.910367 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.910441 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2025094}
I20240503 02:12:41.910773 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 795) took 2032 ms. Trace:
I20240503 02:12:41.910838 5159 rpcz_store.cc:276] 0503 02:12:39.878617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.883316 (+ 4699us) service_pool.cc:225] Handling call
0503 02:12:41.910692 (+2027376us) ts_tablet_manager.cc:1120] Deleting tablet 44707b6439b141b7b63f49f34f25b37a
0503 02:12:41.910698 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.910759 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2025312}
I20240503 02:12:41.911113 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 797) took 2016 ms. Trace:
I20240503 02:12:41.911175 5159 rpcz_store.cc:276] 0503 02:12:39.894851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.901863 (+ 7012us) service_pool.cc:225] Handling call
0503 02:12:41.911021 (+2009158us) ts_tablet_manager.cc:1120] Deleting tablet 4fb59bbf7b68431c86d88f27ef2528b0
0503 02:12:41.911029 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.911098 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008939}
I20240503 02:12:41.911471 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 800) took 1958 ms. Trace:
I20240503 02:12:41.911536 5159 rpcz_store.cc:276] 0503 02:12:39.952535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.952591 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:41.911372 (+1958781us) ts_tablet_manager.cc:1120] Deleting tablet 95c7e4a2954e4161b9acc029873f923b
0503 02:12:41.911382 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.911454 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958622}
I20240503 02:12:41.911696 4811 raft_consensus.cc:2454] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.911805 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 799) took 1963 ms. Trace:
I20240503 02:12:41.911872 5159 rpcz_store.cc:276] 0503 02:12:39.948272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.001120 (+ 52848us) service_pool.cc:225] Handling call
0503 02:12:41.911715 (+1910595us) ts_tablet_manager.cc:1120] Deleting tablet 41531ddcbf66465094d8f06e1116b3be
0503 02:12:41.911725 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.911791 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910307}
I20240503 02:12:41.912101 5159 tablet_replica.cc:330] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:41.913022 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 53017352b8f34955b116bbfd32f04854 because the tablet was not found. No further retry: Not found: Tablet not found: 53017352b8f34955b116bbfd32f04854
I20240503 02:12:41.912569 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:41.913192 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:41.913386 4811 raft_consensus.cc:2454] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:41.913462 4529 raft_consensus.cc:3037] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.913722 5159 raft_consensus.cc:2229] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:41.914093 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 885129a4efc5475b97b7ac68e1db5e1d because the tablet was not found. No further retry: Not found: Tablet not found: 885129a4efc5475b97b7ac68e1db5e1d
W20240503 02:12:41.914822 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2152f205cf1b4ec7ae03605bbfa007e1 because the tablet was not found. No further retry: Not found: Tablet not found: 2152f205cf1b4ec7ae03605bbfa007e1
I20240503 02:12:41.913892 5159 raft_consensus.cc:2258] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:41.915679 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5f56701ed6424c60bf1a985eea1936fe because the tablet was not found. No further retry: Not found: Tablet not found: 5f56701ed6424c60bf1a985eea1936fe
I20240503 02:12:41.915706 5159 ts_tablet_manager.cc:1875] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:41.916366 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 29e5c40f1f6b40308eed26b64d047494 because the tablet was not found. No further retry: Not found: Tablet not found: 29e5c40f1f6b40308eed26b64d047494
I20240503 02:12:41.916536 4529 raft_consensus.cc:2454] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:41.917414 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 41ef3c6e5e7d40249b112144975ba220 because the tablet was not found. No further retry: Not found: Tablet not found: 41ef3c6e5e7d40249b112144975ba220
W20240503 02:12:41.918102 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9fa3df92b6e544de9ca76d98706eebec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa3df92b6e544de9ca76d98706eebec already in progress: creating tablet
W20240503 02:12:41.918931 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ab093285c39b4ce7a0bb3de9d016a918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab093285c39b4ce7a0bb3de9d016a918 already in progress: creating tablet
I20240503 02:12:41.919672 4608 leader_election.cc:302] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:41.920022 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 44707b6439b141b7b63f49f34f25b37a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44707b6439b141b7b63f49f34f25b37a already in progress: creating tablet
W20240503 02:12:41.920975 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4fb59bbf7b68431c86d88f27ef2528b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fb59bbf7b68431c86d88f27ef2528b0 already in progress: creating tablet
W20240503 02:12:41.922214 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 95c7e4a2954e4161b9acc029873f923b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c7e4a2954e4161b9acc029873f923b already in progress: creating tablet
I20240503 02:12:41.923650 5105 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.923826 5105 ts_tablet_manager.cc:1367] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
W20240503 02:12:41.924741 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 41531ddcbf66465094d8f06e1116b3be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41531ddcbf66465094d8f06e1116b3be already in progress: creating tablet
I20240503 02:12:41.924747 5105 raft_consensus.cc:348] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.926635 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9511e4de6c942b9b9e0790c88d3d543" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:41.926856 4811 raft_consensus.cc:3037] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.926906 6579 raft_consensus.cc:2781] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.927058 6579 raft_consensus.cc:686] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:41.927376 6579 consensus_queue.cc:237] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.930104 4811 raft_consensus.cc:2454] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:41.925098 5105 raft_consensus.cc:374] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.930497 5105 raft_consensus.cc:729] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.930778 5105 consensus_queue.cc:260] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.931517 5105 ts_tablet_manager.cc:1398] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:41.931882 6595 raft_consensus.cc:482] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.932013 6595 raft_consensus.cc:504] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.932626 6595 leader_election.cc:288] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:41.933215 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:41.933490 5105 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.933805 5115 ts_tablet_manager.cc:1888] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:41.933830 4746 leader_election.cc:341] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.933974 5115 log.cc:1196] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/41ef3c6e5e7d40249b112144975ba220
I20240503 02:12:41.934216 4529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:41.934355 5115 ts_tablet_manager.cc:1909] T 41ef3c6e5e7d40249b112144975ba220 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:41.934640 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 705) took 3352 ms. Trace:
I20240503 02:12:41.934888 5105 tablet_bootstrap.cc:654] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.937166 5115 rpcz_store.cc:276] 0503 02:12:38.581824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.581876 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:41.776806 (+3194930us) ts_tablet_manager.cc:1120] Deleting tablet 41ef3c6e5e7d40249b112144975ba220
0503 02:12:41.776818 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.843221 (+ 66403us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.933695 (+ 90474us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.934625 (+ 930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3194724}
W20240503 02:12:41.934896 4746 leader_election.cc:341] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:41.937834 5115 tablet_replica.cc:330] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:41.937788 4746 leader_election.cc:302] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:41.938081 5115 raft_consensus.cc:2229] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.938278 5115 raft_consensus.cc:2258] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.938393 6595 raft_consensus.cc:2726] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:41.938860 5115 ts_tablet_manager.cc:1875] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.943205 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 41ef3c6e5e7d40249b112144975ba220 (table default.Table_cf42415d1d604ff283f76a5ac98e7347 [id=ccf48e8d68664a74bb34e616a3715071]) successfully deleted
I20240503 02:12:41.944183 5159 ts_tablet_manager.cc:1888] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.949242 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet d99a7e930a3947b4aae62424f11d7227 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:41.949760 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea11546176f94074a18b91d528290e72 consensus state has no leader
I20240503 02:12:41.949996 5105 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.950325 5105 ts_tablet_manager.cc:1367] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
W20240503 02:12:41.950366 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf091eb225ec4fa88264d438ac95dd59 has no consensus state
I20240503 02:12:41.951269 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:41.951418 5105 raft_consensus.cc:348] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.952071 5105 raft_consensus.cc:374] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.952275 5105 raft_consensus.cc:729] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.952710 5105 consensus_queue.cc:260] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.953580 5159 log.cc:1196] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8a90ebcf29004d1a925b69fa20998dd3
I20240503 02:12:41.954696 5105 ts_tablet_manager.cc:1398] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:41.955135 5159 ts_tablet_manager.cc:1909] T 8a90ebcf29004d1a925b69fa20998dd3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:41.955789 5105 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.957846 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 807) took 1949 ms. Trace:
I20240503 02:12:41.958207 5159 rpcz_store.cc:276] 0503 02:12:40.008162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.008212 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:41.912058 (+1903846us) ts_tablet_manager.cc:1120] Deleting tablet 8a90ebcf29004d1a925b69fa20998dd3
0503 02:12:41.912066 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.941033 (+ 28967us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.944067 (+ 3034us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.957744 (+ 13677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903643}
I20240503 02:12:41.959132 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8a90ebcf29004d1a925b69fa20998dd3 (table default.Table_114d4d6efe8947578cff18d0c2f235d2 [id=41995c76abc744c1b75efd4e1a4b4383]) successfully deleted
I20240503 02:12:41.959499 5159 tablet_replica.cc:330] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:41.960227 5159 raft_consensus.cc:2229] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:41.960780 5159 raft_consensus.cc:2258] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:41.961128 5105 tablet_bootstrap.cc:654] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.962908 5159 ts_tablet_manager.cc:1875] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:41.981220 5159 ts_tablet_manager.cc:1888] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:41.981822 5159 log.cc:1196] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ea5e4db5204e4acf8719100d02d533fb
I20240503 02:12:41.982962 5159 ts_tablet_manager.cc:1909] T ea5e4db5204e4acf8719100d02d533fb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:41.985652 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 808) took 1976 ms. Trace:
I20240503 02:12:41.986016 5105 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:41.986269 5105 ts_tablet_manager.cc:1367] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20240503 02:12:41.987011 5105 raft_consensus.cc:348] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.987305 5105 raft_consensus.cc:374] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.987396 5105 raft_consensus.cc:729] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.987701 5105 consensus_queue.cc:260] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:41.995573 5105 ts_tablet_manager.cc:1398] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:12:41.995985 5105 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:41.985896 5159 rpcz_store.cc:276] 0503 02:12:40.008801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.008848 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:41.959357 (+1950509us) ts_tablet_manager.cc:1120] Deleting tablet ea5e4db5204e4acf8719100d02d533fb
0503 02:12:41.959368 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.969619 (+ 10251us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.980827 (+ 11208us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.985602 (+ 4775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1950345}
I20240503 02:12:41.999076 4528 raft_consensus.cc:3037] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:41.999222 4608 consensus_peers.cc:485] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87713e899d2148308316d807a6d76921. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.006337 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ea5e4db5204e4acf8719100d02d533fb (table default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]) successfully deleted
I20240503 02:12:42.007114 5159 tablet_replica.cc:330] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.007148 6374 raft_consensus.cc:482] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.007386 5159 raft_consensus.cc:2229] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.007555 5159 raft_consensus.cc:2258] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.007419 6374 raft_consensus.cc:504] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.008073 5159 ts_tablet_manager.cc:1875] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.008467 6374 leader_election.cc:288] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.009310 5105 tablet_bootstrap.cc:654] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.012228 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.012449 4811 raft_consensus.cc:2454] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.013069 4608 leader_election.cc:302] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:42.013890 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.014097 6317 raft_consensus.cc:2781] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.014235 6317 raft_consensus.cc:482] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.014353 6317 raft_consensus.cc:3037] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.013612 6579 consensus_queue.cc:1035] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.018187 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 732) took 2627 ms. Trace:
I20240503 02:12:42.018374 5114 rpcz_store.cc:276] 0503 02:12:39.391073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.391127 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:41.746862 (+2355735us) ts_tablet_manager.cc:1120] Deleting tablet 6c9090b11c8a4bf0b4b92523b9431b79
0503 02:12:41.746869 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.777149 (+ 30280us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:41.780027 (+ 2878us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.018163 (+238136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2355560}
I20240503 02:12:42.019038 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 730) took 2629 ms. Trace:
I20240503 02:12:42.019173 5114 rpcz_store.cc:276] 0503 02:12:39.389381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.393887 (+ 4506us) service_pool.cc:225] Handling call
0503 02:12:42.018912 (+2625025us) ts_tablet_manager.cc:1120] Deleting tablet 868a6ad6cc1a40f091b4516676c24c97
0503 02:12:42.018924 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.019021 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2624819}
W20240503 02:12:42.019359 4462 consensus_peers.cc:485] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e713ee37d042309166b0f118e14986. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.019515 5114 tablet_replica.cc:330] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.019749 5114 raft_consensus.cc:2229] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.019984 5114 raft_consensus.cc:2258] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.020634 5114 ts_tablet_manager.cc:1875] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.021826 6317 raft_consensus.cc:504] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.022539 6317 leader_election.cc:288] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.023579 6626 consensus_queue.cc:1035] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.026512 5159 ts_tablet_manager.cc:1888] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:42.026906 4608 leader_election.cc:341] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.027084 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.027683 4811 raft_consensus.cc:3037] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.030347 4811 raft_consensus.cc:2454] T de4e96b4be094101bf74904b2d66b6c7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:42.032140 5159 log.cc:1196] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b9511e4de6c942b9b9e0790c88d3d543
I20240503 02:12:42.032675 5159 ts_tablet_manager.cc:1909] T b9511e4de6c942b9b9e0790c88d3d543 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:42.034054 5106 tablet_bootstrap.cc:492] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.034276 5106 ts_tablet_manager.cc:1367] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.186s user 0.010s sys 0.000s
I20240503 02:12:42.034663 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 809) took 2025 ms. Trace:
I20240503 02:12:42.034981 5159 rpcz_store.cc:276] 0503 02:12:40.009233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.009275 (+ 42us) service_pool.cc:225] Handling call
0503 02:12:42.007054 (+1997779us) ts_tablet_manager.cc:1120] Deleting tablet b9511e4de6c942b9b9e0790c88d3d543
0503 02:12:42.007065 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.018121 (+ 11056us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.026394 (+ 8273us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.034644 (+ 8250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997646}
I20240503 02:12:42.035099 5106 raft_consensus.cc:348] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.035399 5106 raft_consensus.cc:374] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.035504 5106 raft_consensus.cc:729] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.035717 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b9511e4de6c942b9b9e0790c88d3d543 (table default.Table_b417b8282cd14a9aa9bcffd17aab880c [id=e22df2e2673c45cbb1b32201a728333c]) successfully deleted
I20240503 02:12:42.035790 5106 consensus_queue.cc:260] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:42.036434 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efa7c0c5fc614c16985008b8fba54bab consensus state has no leader
I20240503 02:12:42.036470 5106 ts_tablet_manager.cc:1398] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:42.036731 5106 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:42.036899 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 478ac0f515b34e829c9a776bc32d875f consensus state has no leader
I20240503 02:12:42.037626 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:42.027796 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4e96b4be094101bf74904b2d66b6c7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:42.048620 6560 raft_consensus.cc:482] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.048797 6560 raft_consensus.cc:504] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.049516 6560 leader_election.cc:288] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:42.050246 4608 leader_election.cc:341] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.050057 4266 catalog_manager.cc:5484] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.050472 4608 leader_election.cc:302] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.050566 4266 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:42.050747 6374 raft_consensus.cc:2781] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.050731 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:42.050933 6374 raft_consensus.cc:686] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:42.050917 4667 raft_consensus.cc:2454] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:42.051164 6374 consensus_queue.cc:237] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.052037 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.052285 4811 raft_consensus.cc:2454] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:42.053354 4462 leader_election.cc:302] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.054867 6616 raft_consensus.cc:2781] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.055023 6616 raft_consensus.cc:482] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.055194 6616 raft_consensus.cc:3037] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.055155 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 6c9090b11c8a4bf0b4b92523b9431b79 (table default.Table_2c9a62a49c0b48ef86699496645d06df [id=5283ee29bf5142f0928e9aa919fb924a]) successfully deleted
I20240503 02:12:42.055940 4789 tablet_service.cc:1467] Processing CreateTablet for tablet ffef15b7e51840fea91b5349a1724cc5 (DEFAULT_TABLE table=default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b [id=9af1bff5506e47eca463034f1cec98f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.056376 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffef15b7e51840fea91b5349a1724cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.056334 4776 tablet_service.cc:1467] Processing CreateTablet for tablet b391773523fb4880bc788f145ec0b607 (DEFAULT_TABLE table=default.Table_f9386d5ccb884c81a0146719976d0b36 [id=be94555ad7be4061926cee758fc13124]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.056699 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b391773523fb4880bc788f145ec0b607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.056931 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 9db1cca7d1a540b58836fedb2ca1f322 (DEFAULT_TABLE table=default.Table_5468ef29b1e94242b3bb4d5dadf22680 [id=fab5fa7ceea448b392c0ad3e18ac9caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.057227 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db1cca7d1a540b58836fedb2ca1f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.057152 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c27aba1c4782b1d5a87682cf8118 (DEFAULT_TABLE table=default.Table_b591bdbd9f5244d1990cce7fe667b7a5 [id=524679636c474f2ba4d37cb061b2158c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.057435 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c27aba1c4782b1d5a87682cf8118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.058248 6616 raft_consensus.cc:504] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.060572 4497 tablet_service.cc:1467] Processing CreateTablet for tablet ffef15b7e51840fea91b5349a1724cc5 (DEFAULT_TABLE table=default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b [id=9af1bff5506e47eca463034f1cec98f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.060963 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffef15b7e51840fea91b5349a1724cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.061143 4783 tablet_service.cc:1467] Processing CreateTablet for tablet f3b880c9f7ff4874a8e4c342477791ff (DEFAULT_TABLE table=default.Table_55f57cbbc07a4e969857fcb1074dfa54 [id=feba05abc8e642899d584a8271000ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.061477 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b880c9f7ff4874a8e4c342477791ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.065035 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.065243 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.066865 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 9db1cca7d1a540b58836fedb2ca1f322 (DEFAULT_TABLE table=default.Table_5468ef29b1e94242b3bb4d5dadf22680 [id=fab5fa7ceea448b392c0ad3e18ac9caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.067215 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db1cca7d1a540b58836fedb2ca1f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.069212 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca08877cd948e1b04736b54bc779db with delete_type TABLET_DATA_DELETED (replaced by tablet a5f8d331a718475a9f56dc7f3ec0e6b9) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.069638 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.070135 4493 tablet_service.cc:1467] Processing CreateTablet for tablet b391773523fb4880bc788f145ec0b607 (DEFAULT_TABLE table=default.Table_f9386d5ccb884c81a0146719976d0b36 [id=be94555ad7be4061926cee758fc13124]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.070542 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b391773523fb4880bc788f145ec0b607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.070472 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be851867e140416c8921098eaa348201" } modify_external_catalogs: true
I20240503 02:12:42.070709 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_be851867e140416c8921098eaa348201" } modify_external_catalogs: true
I20240503 02:12:42.072182 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2e62fd1e33c643c98023e90fbb294162"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.072803 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e62fd1e33c643c98023e90fbb294162 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.073112 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e474067bd616474aa34cb102a02b6b4d" } modify_external_catalogs: true
I20240503 02:12:42.073469 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_15da6a828615427d9059c958249befbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.073926 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15da6a828615427d9059c958249befbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.073931 5115 ts_tablet_manager.cc:1888] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.074051 5115 log.cc:1196] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/29e5c40f1f6b40308eed26b64d047494
I20240503 02:12:42.074249 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c2cccebe93474d93903c9085efb5a837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.074651 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2cccebe93474d93903c9085efb5a837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.077333 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c27aba1c4782b1d5a87682cf8118 (DEFAULT_TABLE table=default.Table_b591bdbd9f5244d1990cce7fe667b7a5 [id=524679636c474f2ba4d37cb061b2158c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.077670 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c27aba1c4782b1d5a87682cf8118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.082262 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_74c9161da1eb4a9ba2e51a7c8e4b2939" } new_table_name: "default.Table_e7f9cb246f884ebdb6313fdd7c7139f7" modify_external_catalogs: true
I20240503 02:12:42.083673 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e474067bd616474aa34cb102a02b6b4d" } modify_external_catalogs: true
I20240503 02:12:42.084084 4491 tablet_service.cc:1467] Processing CreateTablet for tablet f3b880c9f7ff4874a8e4c342477791ff (DEFAULT_TABLE table=default.Table_55f57cbbc07a4e969857fcb1074dfa54 [id=feba05abc8e642899d584a8271000ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.084455 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b880c9f7ff4874a8e4c342477791ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.091342 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:42.100409 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.100541 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.100680 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.101236 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f3b793f1555b46458c1b7bd8682deeea" } modify_external_catalogs: true
I20240503 02:12:42.101452 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f3b793f1555b46458c1b7bd8682deeea" } modify_external_catalogs: true
W20240503 02:12:42.102759 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 52297.2us
I20240503 02:12:42.114877 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac2ded096f64be0ab4997fb9d09ae84 (DEFAULT_TABLE table=default.Table_0b8cca9f2a7641e6b27283f7a08f0dfe [id=2eb2072ef1dc4438845b67641d6cfc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.114981 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 03fc31800276415fa68d95339025a55f (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.115155 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 47ede57814354d54b3780f1d0ddaa684 (DEFAULT_TABLE table=default.Table_afc821e2c5794ea190f4aec89a0f4676 [id=62cf3c04dff84a409059294784c519d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.115340 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac2ded096f64be0ab4997fb9d09ae84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.114993 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 829adc2091ef4ef6952044baf58c5a81 (DEFAULT_TABLE table=default.Table_742dacc5e26a41fe86b410fe0cfb5b19 [id=4be225c50846426fbab692c5b8cf0733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.115550 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 595728fe05ce4d8f914a1021fe171580 (DEFAULT_TABLE table=default.Table_5303be0a72be4225aeced423b445783c [id=e4fd019baa19407ba180561a673620fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.115736 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829adc2091ef4ef6952044baf58c5a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.115906 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fc31800276415fa68d95339025a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.116151 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 67f9744e82e64adb81c866bdf881be76 (DEFAULT_TABLE table=default.Table_6ded21f98c38492387fbc294230e6de9 [id=a951e2aee80f4e039b8cb5e9e843bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.116436 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f9744e82e64adb81c866bdf881be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.121040 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595728fe05ce4d8f914a1021fe171580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.121274 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ede57814354d54b3780f1d0ddaa684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.131642 6579 raft_consensus.cc:482] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.131807 6579 raft_consensus.cc:504] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.132453 6579 leader_election.cc:288] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.132807 5103 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.132853 4639 tablet_service.cc:1467] Processing CreateTablet for tablet ffef15b7e51840fea91b5349a1724cc5 (DEFAULT_TABLE table=default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b [id=9af1bff5506e47eca463034f1cec98f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.133253 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffef15b7e51840fea91b5349a1724cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.136045 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 829adc2091ef4ef6952044baf58c5a81 (DEFAULT_TABLE table=default.Table_742dacc5e26a41fe86b410fe0cfb5b19 [id=4be225c50846426fbab692c5b8cf0733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.136467 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829adc2091ef4ef6952044baf58c5a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.136586 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 9db1cca7d1a540b58836fedb2ca1f322 (DEFAULT_TABLE table=default.Table_5468ef29b1e94242b3bb4d5dadf22680 [id=fab5fa7ceea448b392c0ad3e18ac9caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.136938 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db1cca7d1a540b58836fedb2ca1f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.144296 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac2ded096f64be0ab4997fb9d09ae84 (DEFAULT_TABLE table=default.Table_0b8cca9f2a7641e6b27283f7a08f0dfe [id=2eb2072ef1dc4438845b67641d6cfc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.144549 5159 tablet_replica.cc:330] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.144737 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac2ded096f64be0ab4997fb9d09ae84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.144780 5159 raft_consensus.cc:2229] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.144985 5159 raft_consensus.cc:2258] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.145555 5159 ts_tablet_manager.cc:1875] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.153237 5103 ts_tablet_manager.cc:1367] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.332s user 0.001s sys 0.005s
I20240503 02:12:42.153957 4645 tablet_service.cc:1467] Processing CreateTablet for tablet b391773523fb4880bc788f145ec0b607 (DEFAULT_TABLE table=default.Table_f9386d5ccb884c81a0146719976d0b36 [id=be94555ad7be4061926cee758fc13124]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.154397 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b391773523fb4880bc788f145ec0b607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.161720 5159 ts_tablet_manager.cc:1888] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:42.162684 5159 log.cc:1196] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:42.163009 5159 ts_tablet_manager.cc:1909] T 035216c9475d4b59aa1ac81b285b4d8a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:42.165120 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 810) took 2155 ms. Trace:
I20240503 02:12:42.165151 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 47ede57814354d54b3780f1d0ddaa684 (DEFAULT_TABLE table=default.Table_afc821e2c5794ea190f4aec89a0f4676 [id=62cf3c04dff84a409059294784c519d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.165223 5159 rpcz_store.cc:276] 0503 02:12:40.009637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.009688 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:42.144488 (+2134800us) ts_tablet_manager.cc:1120] Deleting tablet 035216c9475d4b59aa1ac81b285b4d8a
0503 02:12:42.144500 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.148525 (+ 4025us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.161607 (+ 13082us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.165095 (+ 3488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2134673}
I20240503 02:12:42.165571 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ede57814354d54b3780f1d0ddaa684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.165724 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 035216c9475d4b59aa1ac81b285b4d8a (table default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]) successfully deleted
I20240503 02:12:42.166880 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 811) took 2156 ms. Trace:
I20240503 02:12:42.166973 5159 rpcz_store.cc:276] 0503 02:12:40.010066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.010106 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:42.166763 (+2156657us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:42.166777 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.166860 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2156519}
W20240503 02:12:42.167425 4259 catalog_manager.cc:4797] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f146f7cce0b24d3087bfe0973e2b7ef0 because the tablet was not found. No further retry: Not found: Tablet not found: f146f7cce0b24d3087bfe0973e2b7ef0
I20240503 02:12:42.168258 5159 tablet_replica.cc:330] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.168480 5159 raft_consensus.cc:2229] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.168655 5159 raft_consensus.cc:2258] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.170536 5159 ts_tablet_manager.cc:1875] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.172510 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c27aba1c4782b1d5a87682cf8118 (DEFAULT_TABLE table=default.Table_b591bdbd9f5244d1990cce7fe667b7a5 [id=524679636c474f2ba4d37cb061b2158c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.172910 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c27aba1c4782b1d5a87682cf8118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.177516 4647 tablet_service.cc:1467] Processing CreateTablet for tablet f3b880c9f7ff4874a8e4c342477791ff (DEFAULT_TABLE table=default.Table_55f57cbbc07a4e969857fcb1074dfa54 [id=feba05abc8e642899d584a8271000ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.177917 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b880c9f7ff4874a8e4c342477791ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.178164 5103 raft_consensus.cc:348] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.178390 5103 raft_consensus.cc:374] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.178443 5103 raft_consensus.cc:729] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.178648 5103 consensus_queue.cc:260] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.179162 5103 ts_tablet_manager.cc:1398] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:12:42.180709 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 67f9744e82e64adb81c866bdf881be76 (DEFAULT_TABLE table=default.Table_6ded21f98c38492387fbc294230e6de9 [id=a951e2aee80f4e039b8cb5e9e843bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.181077 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f9744e82e64adb81c866bdf881be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.182713 5103 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.182742 5159 ts_tablet_manager.cc:1888] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:42.182857 5159 log.cc:1196] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/43119471f91c4a1ba900f267c88db9a6
I20240503 02:12:42.183168 5159 ts_tablet_manager.cc:1909] T 43119471f91c4a1ba900f267c88db9a6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:42.183950 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 812) took 2173 ms. Trace:
I20240503 02:12:42.184039 5159 rpcz_store.cc:276] 0503 02:12:40.010570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.010638 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:42.168204 (+2157566us) ts_tablet_manager.cc:1120] Deleting tablet 43119471f91c4a1ba900f267c88db9a6
0503 02:12:42.168215 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.175935 (+ 7720us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.182632 (+ 6697us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.183925 (+ 1293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157430}
I20240503 02:12:42.184245 5103 tablet_bootstrap.cc:654] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.184324 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 829adc2091ef4ef6952044baf58c5a81 (DEFAULT_TABLE table=default.Table_742dacc5e26a41fe86b410fe0cfb5b19 [id=4be225c50846426fbab692c5b8cf0733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.184592 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 43119471f91c4a1ba900f267c88db9a6 (table default.Table_5e542f394cce413f99f8012fcc9fba9a [id=aef6e606e6f34ed78039dd362329ce77]) successfully deleted
I20240503 02:12:42.184695 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829adc2091ef4ef6952044baf58c5a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.193821 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac2ded096f64be0ab4997fb9d09ae84 (DEFAULT_TABLE table=default.Table_0b8cca9f2a7641e6b27283f7a08f0dfe [id=2eb2072ef1dc4438845b67641d6cfc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.194259 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac2ded096f64be0ab4997fb9d09ae84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.197348 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1a082f4c43fd4c188be9ee3966e26edf
I20240503 02:12:42.198225 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3b36c6850376429a8ac96f0a2595db44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.198686 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_230a1a13b4584a188a4235462025db5e" } modify_external_catalogs: true
W20240503 02:12:42.198853 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b36c6850376429a8ac96f0a2595db44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.198904 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_230a1a13b4584a188a4235462025db5e" } modify_external_catalogs: true
I20240503 02:12:42.200209 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet d99a7e930a3947b4aae62424f11d7227
I20240503 02:12:42.200337 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 42e80cc31f5c4908b82f2b030255f6af
I20240503 02:12:42.200733 4269 catalog_manager.cc:6554] ReplaceTablet: tablet d99a7e930a3947b4aae62424f11d7227 deleted and replaced by tablet 452d494e14c440e6805055ba333b4e7f
I20240503 02:12:42.201129 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c535e618b79044af988f914e012015f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.201205 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a341379382784488afd99fb235b712f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.201366 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 47ede57814354d54b3780f1d0ddaa684 (DEFAULT_TABLE table=default.Table_afc821e2c5794ea190f4aec89a0f4676 [id=62cf3c04dff84a409059294784c519d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:42.201635 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c535e618b79044af988f914e012015f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:42.201728 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a341379382784488afd99fb235b712f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.201787 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ede57814354d54b3780f1d0ddaa684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.203142 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e28d38896e194e4eb84ca9cefc1b9835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.203650 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e28d38896e194e4eb84ca9cefc1b9835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.204025 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b7d0640cea2a482e9974376ba479163b
I20240503 02:12:42.204768 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 810d37e454cb4633a099c546925eb48c
I20240503 02:12:42.204793 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1abeaf665dc64f07820a1729b1a0e01d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.205308 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1abeaf665dc64f07820a1729b1a0e01d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.205394 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 73198ec769a9486da3f85e8fcbef8e78 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:42.205948 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_0d42c9ff3ee949f09e9bd50c70d99727" } new_table_name: "default.Table_d68d2154b5174e71a0f8cab394dce18a" modify_external_catalogs: true
I20240503 02:12:42.207428 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7b0e58e30571429b92a41817909e236f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.207882 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b0e58e30571429b92a41817909e236f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:42.209245 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2c43838340e44ef92e318a969e34af6 consensus state has no leader
I20240503 02:12:42.209604 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cfe74b06168d471c86ca7d281c6b2121" } new_table_name: "default.Table_a970660694dc4ff5b2a356e5d36d2beb" modify_external_catalogs: true
I20240503 02:12:42.209887 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 595728fe05ce4d8f914a1021fe171580 (DEFAULT_TABLE table=default.Table_5303be0a72be4225aeced423b445783c [id=e4fd019baa19407ba180561a673620fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.210286 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595728fe05ce4d8f914a1021fe171580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.213618 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.213842 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet d99a7e930a3947b4aae62424f11d7227 with delete_type TABLET_DATA_DELETED (replaced by tablet 452d494e14c440e6805055ba333b4e7f) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.213990 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e80cc31f5c4908b82f2b030255f6af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.215118 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.215337 4811 raft_consensus.cc:2454] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.217417 4608 leader_election.cc:302] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:42.217731 4608 consensus_peers.cc:485] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5e4db5204e4acf8719100d02d533fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.217878 6374 raft_consensus.cc:2781] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.217937 6374 raft_consensus.cc:482] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.217993 6374 raft_consensus.cc:3037] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.218348 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.218703 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e80cc31f5c4908b82f2b030255f6af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.218878 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet d99a7e930a3947b4aae62424f11d7227 with delete_type TABLET_DATA_DELETED (replaced by tablet 452d494e14c440e6805055ba333b4e7f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.219524 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d0640cea2a482e9974376ba479163b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.219905 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 810d37e454cb4633a099c546925eb48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.220594 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.222923 4268 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:42.224208 6374 raft_consensus.cc:504] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.224911 6374 leader_election.cc:288] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.233124 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:42.233935 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b01907833d44f6a000a4de3a2a6e96" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.234127 4811 raft_consensus.cc:3037] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.232723 4268 catalog_manager.cc:5484] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.234670 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:42.235715 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 03fc31800276415fa68d95339025a55f (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.236100 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fc31800276415fa68d95339025a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.236804 4811 raft_consensus.cc:2454] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:42.238548 4608 leader_election.cc:341] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.238919 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 03fc31800276415fa68d95339025a55f (DEFAULT_TABLE table=default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.239322 5159 tablet_replica.cc:330] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.239413 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fc31800276415fa68d95339025a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.239794 5159 raft_consensus.cc:2229] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.240272 5159 raft_consensus.cc:2258] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.241081 5159 ts_tablet_manager.cc:1875] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.246013 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet d99a7e930a3947b4aae62424f11d7227 with delete_type TABLET_DATA_DELETED (replaced by tablet 452d494e14c440e6805055ba333b4e7f) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.246413 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.246515 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e80cc31f5c4908b82f2b030255f6af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.246879 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d0640cea2a482e9974376ba479163b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.247222 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 810d37e454cb4633a099c546925eb48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.253415 5115 ts_tablet_manager.cc:1909] T 29e5c40f1f6b40308eed26b64d047494 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.253827 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 706) took 3671 ms. Trace:
I20240503 02:12:42.253929 5115 rpcz_store.cc:276] 0503 02:12:38.582289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.582334 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:41.937768 (+3355434us) ts_tablet_manager.cc:1120] Deleting tablet 29e5c40f1f6b40308eed26b64d047494
0503 02:12:41.937783 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:41.988164 (+ 50381us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.073820 (+ 85656us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.253806 (+179986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3355273}
I20240503 02:12:42.254249 5115 tablet_replica.cc:330] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.254429 5115 raft_consensus.cc:2229] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.254397 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 595728fe05ce4d8f914a1021fe171580 (DEFAULT_TABLE table=default.Table_5303be0a72be4225aeced423b445783c [id=e4fd019baa19407ba180561a673620fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.254580 5115 raft_consensus.cc:2258] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.254809 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595728fe05ce4d8f914a1021fe171580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.254942 5105 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:42.255079 5105 ts_tablet_manager.cc:1367] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.259s user 0.016s sys 0.016s
I20240503 02:12:42.255097 5115 ts_tablet_manager.cc:1875] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.255787 5105 raft_consensus.cc:348] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.255975 5105 raft_consensus.cc:374] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.256072 5105 raft_consensus.cc:729] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.256304 5105 consensus_queue.cc:260] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.257917 6317 raft_consensus.cc:482] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.258044 6317 raft_consensus.cc:504] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.258658 6317 leader_election.cc:288] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:42.238977 4608 leader_election.cc:341] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.260871 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.261070 4811 raft_consensus.cc:2454] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.261183 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 29e5c40f1f6b40308eed26b64d047494 (table default.Table_464b5c2e54804303a2039f77c6a57dd4 [id=e9bf79d08dc04dc9a4e5739dea750285]) successfully deleted
I20240503 02:12:42.261766 6317 raft_consensus.cc:482] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.261826 5115 ts_tablet_manager.cc:1888] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.261914 5115 log.cc:1196] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3b72c20a4b5a4d2ea266f031733d1aab
I20240503 02:12:42.261879 6317 raft_consensus.cc:504] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.262037 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.262245 5115 ts_tablet_manager.cc:1909] T 3b72c20a4b5a4d2ea266f031733d1aab P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.262236 4528 raft_consensus.cc:2454] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.262497 6317 leader_election.cc:288] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.262521 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 707) took 3679 ms. Trace:
I20240503 02:12:42.262581 5115 rpcz_store.cc:276] 0503 02:12:38.582653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.582692 (+ 39us) service_pool.cc:225] Handling call
0503 02:12:42.254205 (+3671513us) ts_tablet_manager.cc:1120] Deleting tablet 3b72c20a4b5a4d2ea266f031733d1aab
0503 02:12:42.254214 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.258827 (+ 4613us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.261734 (+ 2907us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.262504 (+ 770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3671382}
I20240503 02:12:42.262938 5115 tablet_replica.cc:330] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.263157 5115 raft_consensus.cc:2229] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.263327 5115 raft_consensus.cc:2258] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.263829 5115 ts_tablet_manager.cc:1875] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.265511 4608 leader_election.cc:302] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.265767 6317 raft_consensus.cc:2781] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.265861 6317 raft_consensus.cc:686] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:42.266057 6317 consensus_queue.cc:237] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.269608 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 810d37e454cb4633a099c546925eb48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.273126 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d0640cea2a482e9974376ba479163b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.276152 5105 ts_tablet_manager.cc:1398] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 02:12:42.276566 5115 ts_tablet_manager.cc:1888] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:42.277921 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.278018 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 67f9744e82e64adb81c866bdf881be76 (DEFAULT_TABLE table=default.Table_6ded21f98c38492387fbc294230e6de9 [id=a951e2aee80f4e039b8cb5e9e843bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.278105 4811 raft_consensus.cc:3037] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.278384 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f9744e82e64adb81c866bdf881be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.280675 4811 raft_consensus.cc:2454] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.281239 4462 leader_election.cc:302] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.283561 6620 raft_consensus.cc:2781] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.284286 6620 raft_consensus.cc:686] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:42.284653 6616 leader_election.cc:288] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.284713 6620 consensus_queue.cc:237] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.288174 5105 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:42.294006 5159 ts_tablet_manager.cc:1888] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:42.295220 5159 log.cc:1196] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:42.295643 5159 ts_tablet_manager.cc:1909] T b38d63d2cc8c4f54ba43cdf8cde1841f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:42.294221 5105 tablet_bootstrap.cc:654] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.297206 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 64cdb861bc9f448698e243718136c303
I20240503 02:12:42.298013 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_87ee6bd84c764979a8df9a93bdb3d424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.298221 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cdb861bc9f448698e243718136c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:42.298686 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87ee6bd84c764979a8df9a93bdb3d424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.299325 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 813) took 2288 ms. Trace:
I20240503 02:12:42.300083 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet b5f444b22112407ca02fe86ac50167a9 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:42.300654 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2c9a62a49c0b48ef86699496645d06df" } new_table_name: "default.Table_e6fc3292e398448cbb24c71e0c98ec56" modify_external_catalogs: true
I20240503 02:12:42.304055 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3755b2f2996b4bc785bf23c4c7af2cf2" } new_table_name: "default.Table_2acba999c8d0432caf7f1cc0f404b8d0" modify_external_catalogs: true
I20240503 02:12:42.304869 5159 rpcz_store.cc:276] 0503 02:12:40.011025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.011074 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:42.239265 (+2228191us) ts_tablet_manager.cc:1120] Deleting tablet b38d63d2cc8c4f54ba43cdf8cde1841f
0503 02:12:42.239277 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.248713 (+ 9436us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.293902 (+ 45189us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.299304 (+ 5402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228035}
I20240503 02:12:42.305378 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b38d63d2cc8c4f54ba43cdf8cde1841f (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]) successfully deleted
W20240503 02:12:42.305464 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a86ac17982324847949b8a9245329489 consensus state has no leader
I20240503 02:12:42.306107 5159 tablet_replica.cc:330] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.306308 5159 raft_consensus.cc:2229] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.306464 5159 raft_consensus.cc:2258] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.307194 5159 ts_tablet_manager.cc:1875] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:42.307673 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 061742d380b1496bbfaaa45ead1d0214 consensus state has no leader
I20240503 02:12:42.308180 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet 5a0303f1b1194115a7142a00470457c3 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:42.311647 4462 consensus_peers.cc:485] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95681e67db1e4485876d530133e8a514. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.314087 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.317580 5103 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.317698 5103 ts_tablet_manager.cc:1367] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.002s
I20240503 02:12:42.318490 5103 raft_consensus.cc:348] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.318666 5103 raft_consensus.cc:374] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.318704 5103 raft_consensus.cc:729] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.318882 5103 consensus_queue.cc:260] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.319335 5103 ts_tablet_manager.cc:1398] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:42.321466 5103 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.323011 5103 tablet_bootstrap.cc:654] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.323909 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cdb861bc9f448698e243718136c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:42.325947 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7ba9a44d78b4a649a6d358e5c187974 consensus state has no leader
I20240503 02:12:42.332346 5114 ts_tablet_manager.cc:1888] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.332562 5114 log.cc:1196] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/f2f9b962c9264f87b798df33544cb15d
I20240503 02:12:42.346625 5105 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:42.346890 5105 ts_tablet_manager.cc:1367] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20240503 02:12:42.347783 5105 raft_consensus.cc:348] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.348140 5105 raft_consensus.cc:374] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.348254 5105 raft_consensus.cc:729] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.348619 5105 consensus_queue.cc:260] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.348956 5114 ts_tablet_manager.cc:1909] T f2f9b962c9264f87b798df33544cb15d P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:42.357635 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 972cd318ebed41c0ab1b8705f52bdaa4 consensus state has no leader
I20240503 02:12:42.276762 5106 tablet_bootstrap.cc:654] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.277509 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4972057fae45168756d7c28ddbe2bf" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:42.362999 4675 raft_consensus.cc:3037] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.363863 5103 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.364032 5103 ts_tablet_manager.cc:1367] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.013s
I20240503 02:12:42.365204 5103 raft_consensus.cc:348] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.365561 5103 raft_consensus.cc:374] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.365685 5103 raft_consensus.cc:729] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.365856 4675 raft_consensus.cc:2454] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.365990 5103 consensus_queue.cc:260] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.283618 4608 leader_election.cc:302] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:42.366657 5115 log.cc:1196] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/d271d2c112ba44b68d2933a6ebfe7a18
W20240503 02:12:42.366789 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 83192.5us
I20240503 02:12:42.366954 5115 ts_tablet_manager.cc:1909] T d271d2c112ba44b68d2933a6ebfe7a18 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.367244 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 708) took 3784 ms. Trace:
I20240503 02:12:42.367312 5115 rpcz_store.cc:276] 0503 02:12:38.582966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.583006 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:42.262896 (+3679890us) ts_tablet_manager.cc:1120] Deleting tablet d271d2c112ba44b68d2933a6ebfe7a18
0503 02:12:42.262906 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.273149 (+ 10243us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.276481 (+ 3332us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.367223 (+ 90742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3679742}
I20240503 02:12:42.367615 5115 tablet_replica.cc:330] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.367787 5115 raft_consensus.cc:2229] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:42.367856 6317 raft_consensus.cc:2781] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.367933 6317 raft_consensus.cc:482] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.368026 6317 raft_consensus.cc:3037] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.368041 5115 raft_consensus.cc:2258] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.368584 5115 ts_tablet_manager.cc:1875] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.370393 6374 raft_consensus.cc:482] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:42.370529 6374 raft_consensus.cc:504] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:42.371109 6374 leader_election.cc:288] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:42.371420 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.371486 5103 ts_tablet_manager.cc:1398] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:42.371630 4811 raft_consensus.cc:2454] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.371764 5103 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.371487 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 738) took 2922 ms. Trace:
I20240503 02:12:42.372611 5114 rpcz_store.cc:276] 0503 02:12:39.449036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.449097 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:42.019470 (+2570373us) ts_tablet_manager.cc:1120] Deleting tablet f2f9b962c9264f87b798df33544cb15d
0503 02:12:42.019480 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.023814 (+ 4334us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.332229 (+308415us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.371469 (+ 39240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2570158}
I20240503 02:12:42.373025 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 735) took 2925 ms. Trace:
I20240503 02:12:42.373106 5114 rpcz_store.cc:276] 0503 02:12:39.447900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.450991 (+ 3091us) service_pool.cc:225] Handling call
0503 02:12:42.372912 (+2921921us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:42.372938 (+ 26us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.373006 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921740}
I20240503 02:12:42.373405 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 736) took 2924 ms. Trace:
I20240503 02:12:42.373493 5114 rpcz_store.cc:276] 0503 02:12:39.448528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.451206 (+ 2678us) service_pool.cc:225] Handling call
0503 02:12:42.373327 (+2922121us) ts_tablet_manager.cc:1120] Deleting tablet d022b5df4a914e439fff918f70c34157
0503 02:12:42.373335 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.373387 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921960}
I20240503 02:12:42.373662 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.373783 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 737) took 2925 ms. Trace:
I20240503 02:12:42.373842 4528 raft_consensus.cc:2454] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:42.374841 5103 tablet_bootstrap.cc:654] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.373859 5114 rpcz_store.cc:276] 0503 02:12:39.448771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.451664 (+ 2893us) service_pool.cc:225] Handling call
0503 02:12:42.373706 (+2922042us) ts_tablet_manager.cc:1120] Deleting tablet 5f56701ed6424c60bf1a985eea1936fe
0503 02:12:42.373714 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.373765 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921896}
I20240503 02:12:42.375463 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 741) took 2905 ms. Trace:
I20240503 02:12:42.375528 5114 rpcz_store.cc:276] 0503 02:12:39.469887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.469950 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:42.375359 (+2905409us) ts_tablet_manager.cc:1120] Deleting tablet 1716c4807e004760b34c0492cb7f522d
0503 02:12:42.375367 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.375448 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2905221}
I20240503 02:12:42.375790 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 742) took 2861 ms. Trace:
I20240503 02:12:42.375849 5114 rpcz_store.cc:276] 0503 02:12:39.514146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.515409 (+ 1263us) service_pool.cc:225] Handling call
0503 02:12:42.375707 (+2860298us) ts_tablet_manager.cc:1120] Deleting tablet 5e534255f170416690c18ca6bd571117
0503 02:12:42.375714 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.375777 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2860041}
I20240503 02:12:42.376097 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 743) took 2832 ms. Trace:
I20240503 02:12:42.376161 5114 rpcz_store.cc:276] 0503 02:12:39.544033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.544099 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:42.376018 (+2831919us) ts_tablet_manager.cc:1120] Deleting tablet f20e573d70bd46aab33466ccfc801465
0503 02:12:42.376026 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.376084 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2831658}
I20240503 02:12:42.376394 5114 tablet_replica.cc:330] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.376557 5114 raft_consensus.cc:2229] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.376717 5114 raft_consensus.cc:2258] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.377296 5114 ts_tablet_manager.cc:1875] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.382174 6317 raft_consensus.cc:504] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.382817 6317 leader_election.cc:288] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.383517 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 3b72c20a4b5a4d2ea266f031733d1aab (table default.Table_39060d5d029f4211a2c27b981453e636 [id=5e564c0605794c16ad8b881b01a8e065]) successfully deleted
I20240503 02:12:42.384161 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet f2f9b962c9264f87b798df33544cb15d (table default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]) successfully deleted
W20240503 02:12:42.384380 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 53017352b8f34955b116bbfd32f04854 because the tablet was not found. No further retry: Not found: Tablet not found: 53017352b8f34955b116bbfd32f04854
W20240503 02:12:42.384596 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d022b5df4a914e439fff918f70c34157 because the tablet was not found. No further retry: Not found: Tablet not found: d022b5df4a914e439fff918f70c34157
I20240503 02:12:42.384660 5114 ts_tablet_manager.cc:1888] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.384785 5114 log.cc:1196] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7461aa76a1af41cb9253285bda708f30
W20240503 02:12:42.384778 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5f56701ed6424c60bf1a985eea1936fe because the tablet was not found. No further retry: Not found: Tablet not found: 5f56701ed6424c60bf1a985eea1936fe
W20240503 02:12:42.388283 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1716c4807e004760b34c0492cb7f522d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1716c4807e004760b34c0492cb7f522d already in progress: creating tablet
W20240503 02:12:42.388553 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5e534255f170416690c18ca6bd571117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e534255f170416690c18ca6bd571117 already in progress: creating tablet
W20240503 02:12:42.388761 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f20e573d70bd46aab33466ccfc801465 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20e573d70bd46aab33466ccfc801465 already in progress: creating tablet
I20240503 02:12:42.392148 5103 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.392371 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cdb861bc9f448698e243718136c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.392756 5106 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.392832 4608 leader_election.cc:302] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.392887 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.392964 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4972057fae45168756d7c28ddbe2bf with delete_type TABLET_DATA_DELETED (replaced by tablet 054d57b190ef42ba81f1c8a2cf00761b) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.392881 5106 ts_tablet_manager.cc:1367] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.356s user 0.007s sys 0.001s
I20240503 02:12:42.393488 6638 raft_consensus.cc:2781] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.393568 6638 raft_consensus.cc:482] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.393678 6638 raft_consensus.cc:3037] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.393890 5106 raft_consensus.cc:348] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.394066 5106 raft_consensus.cc:374] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.394119 5106 raft_consensus.cc:729] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.394245 6579 consensus_queue.cc:1035] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.394320 5106 consensus_queue.cc:260] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.395272 5106 ts_tablet_manager.cc:1398] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:42.395869 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.396056 4807 raft_consensus.cc:3037] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.398748 4807 raft_consensus.cc:2454] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:42.399765 6638 raft_consensus.cc:504] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.400952 6638 leader_election.cc:288] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.400161 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c27e37ebd0f49cdba7653548acbe1ea" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:42.401264 4527 raft_consensus.cc:3037] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.401444 5103 ts_tablet_manager.cc:1367] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
W20240503 02:12:42.402035 4608 leader_election.cc:341] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 1c2cdd9fdae843fe86c779e8eb67bfe2
W20240503 02:12:42.402340 4608 consensus_peers.cc:485] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2f9b962c9264f87b798df33544cb15d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.402431 5103 raft_consensus.cc:348] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.403061 5103 raft_consensus.cc:374] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.403120 4608 leader_election.cc:302] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:42.403187 5103 raft_consensus.cc:729] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.403412 5103 consensus_queue.cc:260] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.403589 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.403769 4811 raft_consensus.cc:3037] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.403785 4527 raft_consensus.cc:2454] T 8c27e37ebd0f49cdba7653548acbe1ea P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:42.404332 5103 ts_tablet_manager.cc:1398] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:42.404733 5103 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.406416 4811 raft_consensus.cc:2454] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:42.406661 5103 tablet_bootstrap.cc:654] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:42.407418 4608 leader_election.cc:341] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:42.407735 4608 leader_election.cc:302] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.407572 5114 ts_tablet_manager.cc:1909] T 7461aa76a1af41cb9253285bda708f30 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:42.408461 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca08877cd948e1b04736b54bc779db" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:42.408634 4527 raft_consensus.cc:3037] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.408545 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 748) took 2767 ms. Trace:
I20240503 02:12:42.408687 6262 raft_consensus.cc:2781] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.408804 6262 raft_consensus.cc:686] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:42.408794 5114 rpcz_store.cc:276] 0503 02:12:39.641104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.641162 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:42.376359 (+2735197us) ts_tablet_manager.cc:1120] Deleting tablet 7461aa76a1af41cb9253285bda708f30
0503 02:12:42.376366 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.380360 (+ 3994us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.384562 (+ 4202us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.408530 (+ 23968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2734998}
I20240503 02:12:42.409067 6262 consensus_queue.cc:237] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.409560 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 749) took 2709 ms. Trace:
I20240503 02:12:42.409670 5114 rpcz_store.cc:276] 0503 02:12:39.700434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.700494 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:42.409444 (+2708950us) ts_tablet_manager.cc:1120] Deleting tablet 9fa3df92b6e544de9ca76d98706eebec
0503 02:12:42.409453 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.409546 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2708748}
I20240503 02:12:42.410087 5114 tablet_replica.cc:330] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.410070 6579 raft_consensus.cc:2726] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:42.410339 5114 raft_consensus.cc:2229] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.410537 5114 raft_consensus.cc:2258] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.411098 5114 ts_tablet_manager.cc:1875] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.411384 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]) successfully deleted
I20240503 02:12:42.411427 4527 raft_consensus.cc:2454] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:42.411691 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9fa3df92b6e544de9ca76d98706eebec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa3df92b6e544de9ca76d98706eebec already in progress: creating tablet
I20240503 02:12:42.411916 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet d271d2c112ba44b68d2933a6ebfe7a18 (table default.Table_d54df163e4634a779d3bcfddb083db60 [id=363231755ce84709b926a7416f3bc40a]) successfully deleted
I20240503 02:12:42.412355 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_98865c1dc735482d9fdb50d75f612ac9" } modify_external_catalogs: true
I20240503 02:12:42.412582 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_98865c1dc735482d9fdb50d75f612ac9" } modify_external_catalogs: true
I20240503 02:12:42.413945 6616 raft_consensus.cc:482] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.414070 6616 raft_consensus.cc:504] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.414629 6616 leader_election.cc:288] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.415951 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_81c7b0a5ee98426c9e050f8f5f8d7d35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.416234 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.416468 4811 raft_consensus.cc:2454] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
W20240503 02:12:42.416622 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81c7b0a5ee98426c9e050f8f5f8d7d35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.416880 4462 leader_election.cc:302] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.417261 6616 raft_consensus.cc:2781] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.417337 6616 raft_consensus.cc:482] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.417409 6616 raft_consensus.cc:3037] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.419550 5114 ts_tablet_manager.cc:1888] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.420127 6616 raft_consensus.cc:504] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.422932 6616 leader_election.cc:288] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.423146 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_555e1f783a5f411999e6a287d62c7d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.423420 5114 log.cc:1196] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7394c9d374404fb199ab6ff170d7febe
I20240503 02:12:42.423418 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_817b839a30804e609f4916cf44d4dafd" } modify_external_catalogs: true
I20240503 02:12:42.423606 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_817b839a30804e609f4916cf44d4dafd" } modify_external_catalogs: true
I20240503 02:12:42.423775 5114 ts_tablet_manager.cc:1909] T 7394c9d374404fb199ab6ff170d7febe P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:42.423807 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_555e1f783a5f411999e6a287d62c7d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.423794 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.424027 4811 raft_consensus.cc:3037] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.424141 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_86bc1fe3319346129d38c1c2d731742f" } modify_external_catalogs: true
I20240503 02:12:42.424329 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_86bc1fe3319346129d38c1c2d731742f" } modify_external_catalogs: true
I20240503 02:12:42.424816 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:42.425360 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_997fff88119a49a5b51f25324fe4f546" } modify_external_catalogs: true
W20240503 02:12:42.425477 4462 leader_election.cc:341] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.425607 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_997fff88119a49a5b51f25324fe4f546" } modify_external_catalogs: true
I20240503 02:12:42.425659 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 752) took 2536 ms. Trace:
I20240503 02:12:42.425649 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:42.425726 5114 rpcz_store.cc:276] 0503 02:12:39.889107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.889163 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:42.410044 (+2520881us) ts_tablet_manager.cc:1120] Deleting tablet 7394c9d374404fb199ab6ff170d7febe
0503 02:12:42.410053 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.414997 (+ 4944us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.419443 (+ 4446us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.425641 (+ 6198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2520686}
W20240503 02:12:42.426146 4462 leader_election.cc:341] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.426764 5114 tablet_replica.cc:330] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.427407 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7394c9d374404fb199ab6ff170d7febe (table default.Table_be53cf9a5c3a418d9dc24120c17d3a4b [id=646347328e89442b8713fa69f5078052]) successfully deleted
W20240503 02:12:42.427897 4608 consensus_peers.cc:485] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3db94fb9d53044d796e2973143d888fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.427151 5114 raft_consensus.cc:2229] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.429422 5114 raft_consensus.cc:2258] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.430334 5114 ts_tablet_manager.cc:1875] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.430931 4811 raft_consensus.cc:2454] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.431370 4608 leader_election.cc:302] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:42.433535 6262 raft_consensus.cc:2781] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.433698 6262 raft_consensus.cc:686] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:42.433956 6262 consensus_queue.cc:237] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.436719 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_92682024602d4fdbae60fe91dc9b2651" } new_table_name: "default.Table_ab7e62fbcb4a4d5480734eeaf0de575f" modify_external_catalogs: true
I20240503 02:12:42.441210 5106 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.442703 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1643ac26c14545c49e6ce2196a666402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:42.443125 5106 tablet_bootstrap.cc:654] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:42.443346 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1643ac26c14545c49e6ce2196a666402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.443606 5114 ts_tablet_manager.cc:1888] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.454828 4267 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:42.458298 5114 log.cc:1196] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b91abaf51abb4c2ca65412c3771c1ef2
I20240503 02:12:42.459108 5114 ts_tablet_manager.cc:1909] T b91abaf51abb4c2ca65412c3771c1ef2 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:42.459141 4267 catalog_manager.cc:5484] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.459734 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:42.460718 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 753) took 2571 ms. Trace:
I20240503 02:12:42.461284 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 5a0303f1b1194115a7142a00470457c3
I20240503 02:12:42.461871 4274 catalog_manager.cc:6554] ReplaceTablet: tablet 5a0303f1b1194115a7142a00470457c3 deleted and replaced by tablet cc0ea917e51e48ae8c8395af7ed85cfe
I20240503 02:12:42.462097 5114 rpcz_store.cc:276] 0503 02:12:39.889286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.889407 (+ 121us) service_pool.cc:225] Handling call
0503 02:12:42.426716 (+2537309us) ts_tablet_manager.cc:1120] Deleting tablet b91abaf51abb4c2ca65412c3771c1ef2
0503 02:12:42.426726 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.440514 (+ 13788us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.443513 (+ 2999us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.460696 (+ 17183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2537154}
I20240503 02:12:42.462903 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 755) took 2573 ms. Trace:
I20240503 02:12:42.462985 5114 rpcz_store.cc:276] 0503 02:12:39.889551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.889657 (+ 106us) service_pool.cc:225] Handling call
0503 02:12:42.462800 (+2573143us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:42.462810 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.462884 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2572972}
I20240503 02:12:42.463835 5114 tablet_replica.cc:330] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.464078 5114 raft_consensus.cc:2229] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:42.464439 5114 raft_consensus.cc:2258] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.465415 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0303f1b1194115a7142a00470457c3 with delete_type TABLET_DATA_DELETED (replaced by tablet cc0ea917e51e48ae8c8395af7ed85cfe) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.465638 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0303f1b1194115a7142a00470457c3 with delete_type TABLET_DATA_DELETED (replaced by tablet cc0ea917e51e48ae8c8395af7ed85cfe) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.466121 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b91abaf51abb4c2ca65412c3771c1ef2 (table default.Table_df633d54f27d4e71b052cbf0582d8ba2 [id=65c98763e63d4447b90a16f448bccffe]) successfully deleted
I20240503 02:12:42.466282 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:42.466511 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 65774b69a8bd4664a6b55b84eba37f7f because the tablet was not found. No further retry: Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:42.466905 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 01c61bf67c8a47fb9ef23bcd6c58c448
I20240503 02:12:42.467065 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0303f1b1194115a7142a00470457c3 with delete_type TABLET_DATA_DELETED (replaced by tablet cc0ea917e51e48ae8c8395af7ed85cfe) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.467456 5105 ts_tablet_manager.cc:1398] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.120s user 0.002s sys 0.000s
I20240503 02:12:42.465085 5114 ts_tablet_manager.cc:1875] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.469089 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:42.469964 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.470616 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:42.470510 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.473379 5105 tablet_bootstrap.cc:492] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:42.451267 4462 leader_election.cc:302] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:42.478574 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:42.478996 6616 raft_consensus.cc:2781] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.479125 6616 raft_consensus.cc:686] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:42.479375 6616 consensus_queue.cc:237] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.481125 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6e62283ba040dba09083a86b4516c8 (DEFAULT_TABLE table=default.Table_2843778966d549a18acea39be8ad6929 [id=013401c0654140e1b218a55a15bb1f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.481426 4645 tablet_service.cc:1467] Processing CreateTablet for tablet c8f843a0951a4341bdcbf20cf29a0b25 (DEFAULT_TABLE table=default.Table_7be6ff0537114d788c876ce810d2ae5b [id=7d08a611b2b647e0934a0dc01a653021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.481523 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6e62283ba040dba09083a86b4516c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:42.481685 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb2c0539bc1c4140b09ac51ad670c0e5 consensus state has no leader
I20240503 02:12:42.482197 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6e62283ba040dba09083a86b4516c8 (DEFAULT_TABLE table=default.Table_2843778966d549a18acea39be8ad6929 [id=013401c0654140e1b218a55a15bb1f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.485193 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 7011db39a7fa4b41903784d2c3f0c1f8 (DEFAULT_TABLE table=default.Table_7b24aa2076944be5bc2a5fbf74726758 [id=296c77f35f244f7294cd86e7eda32511]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.485551 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011db39a7fa4b41903784d2c3f0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.479321 4780 tablet_service.cc:1467] Processing CreateTablet for tablet c8f843a0951a4341bdcbf20cf29a0b25 (DEFAULT_TABLE table=default.Table_7be6ff0537114d788c876ce810d2ae5b [id=7d08a611b2b647e0934a0dc01a653021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.487071 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f843a0951a4341bdcbf20cf29a0b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.480114 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6e62283ba040dba09083a86b4516c8 (DEFAULT_TABLE table=default.Table_2843778966d549a18acea39be8ad6929 [id=013401c0654140e1b218a55a15bb1f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.489225 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6e62283ba040dba09083a86b4516c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.492154 4491 tablet_service.cc:1467] Processing CreateTablet for tablet c8f843a0951a4341bdcbf20cf29a0b25 (DEFAULT_TABLE table=default.Table_7be6ff0537114d788c876ce810d2ae5b [id=7d08a611b2b647e0934a0dc01a653021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.480758 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 7011db39a7fa4b41903784d2c3f0c1f8 (DEFAULT_TABLE table=default.Table_7b24aa2076944be5bc2a5fbf74726758 [id=296c77f35f244f7294cd86e7eda32511]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.492537 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f843a0951a4341bdcbf20cf29a0b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.492581 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011db39a7fa4b41903784d2c3f0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.493335 5115 ts_tablet_manager.cc:1888] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.493458 5115 log.cc:1196] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/2152f205cf1b4ec7ae03605bbfa007e1
I20240503 02:12:42.493820 5115 ts_tablet_manager.cc:1909] T 2152f205cf1b4ec7ae03605bbfa007e1 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.494104 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 709) took 3910 ms. Trace:
I20240503 02:12:42.494174 5115 rpcz_store.cc:276] 0503 02:12:38.583321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.583365 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:42.367574 (+3784209us) ts_tablet_manager.cc:1120] Deleting tablet 2152f205cf1b4ec7ae03605bbfa007e1
0503 02:12:42.367582 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.414565 (+ 46983us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.488965 (+ 74400us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.494086 (+ 5121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3784047}
I20240503 02:12:42.494589 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 710) took 3910 ms. Trace:
I20240503 02:12:42.494658 5115 rpcz_store.cc:276] 0503 02:12:38.583703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.583737 (+ 34us) service_pool.cc:225] Handling call
0503 02:12:42.494492 (+3910755us) ts_tablet_manager.cc:1120] Deleting tablet 3e6a3ca705e8482a8cd6a6fb17d93470
0503 02:12:42.494501 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.494573 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3910621}
I20240503 02:12:42.494890 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 711) took 3909 ms. Trace:
I20240503 02:12:42.494941 5115 rpcz_store.cc:276] 0503 02:12:38.585348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.585451 (+ 103us) service_pool.cc:225] Handling call
0503 02:12:42.494835 (+3909384us) ts_tablet_manager.cc:1120] Deleting tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
0503 02:12:42.494841 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.494881 (+ 40us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3909239}
I20240503 02:12:42.495169 5115 tablet_replica.cc:330] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.495370 5115 raft_consensus.cc:2229] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.495543 5115 raft_consensus.cc:2258] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.496071 5115 ts_tablet_manager.cc:1875] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.500236 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f843a0951a4341bdcbf20cf29a0b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.501744 5106 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.501876 5106 ts_tablet_manager.cc:1367] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.002s
I20240503 02:12:42.502768 5106 raft_consensus.cc:348] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.502970 5106 raft_consensus.cc:374] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.503027 5106 raft_consensus.cc:729] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.503224 5106 consensus_queue.cc:260] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.503720 5106 ts_tablet_manager.cc:1398] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:42.504050 5106 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.505239 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6e62283ba040dba09083a86b4516c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.508592 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bdfc07f1b4c94c97b1eddb89795526c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.509253 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdfc07f1b4c94c97b1eddb89795526c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.509601 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c1106a95348a4a3f8961db51ddeb4221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.510130 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1106a95348a4a3f8961db51ddeb4221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.510713 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cec6bc0bc6bc45b495af097e7fdb43bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.511234 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cec6bc0bc6bc45b495af097e7fdb43bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.511317 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_8c58e481444e4e98b065e88e715ba236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:42.511772 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c58e481444e4e98b065e88e715ba236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:42.515131 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c24f2a34ca8a43a388947bcc6a82f54e" } modify_external_catalogs: true
I20240503 02:12:42.515409 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c24f2a34ca8a43a388947bcc6a82f54e" } modify_external_catalogs: true
I20240503 02:12:42.519162 5106 tablet_bootstrap.cc:654] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.527987 5106 tablet_bootstrap.cc:492] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.528240 5106 ts_tablet_manager.cc:1367] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
I20240503 02:12:42.529090 5106 raft_consensus.cc:348] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.529505 5106 raft_consensus.cc:374] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.531342 5105 tablet_bootstrap.cc:654] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.529620 5106 raft_consensus.cc:729] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.532759 5106 consensus_queue.cc:260] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.533576 5103 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.533601 5106 ts_tablet_manager.cc:1398] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:42.533732 5103 ts_tablet_manager.cc:1367] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.011s
I20240503 02:12:42.534070 5106 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.535737 5106 tablet_bootstrap.cc:654] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.536464 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 07a529884fc240da98aa910371610f3a (DEFAULT_TABLE table=default.Table_ebe4e5a3a5124836ab1d84342f373932 [id=b2089db97f8c461eaa1269b37ec8e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.536562 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 07a529884fc240da98aa910371610f3a (DEFAULT_TABLE table=default.Table_ebe4e5a3a5124836ab1d84342f373932 [id=b2089db97f8c461eaa1269b37ec8e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.536993 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a529884fc240da98aa910371610f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.537039 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a529884fc240da98aa910371610f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.540019 5103 raft_consensus.cc:348] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.542035 5103 raft_consensus.cc:374] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.542390 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 7011db39a7fa4b41903784d2c3f0c1f8 (DEFAULT_TABLE table=default.Table_7b24aa2076944be5bc2a5fbf74726758 [id=296c77f35f244f7294cd86e7eda32511]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:42.542977 4608 consensus_peers.cc:485] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b91abaf51abb4c2ca65412c3771c1ef2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.543871 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86814002734d018b2052bdf9bdedab (DEFAULT_TABLE table=default.Table_8de2edcca99e40cdaf5db0ea01f4a652 [id=baa0ab04a63642178ac0b350438c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.544198 5103 raft_consensus.cc:729] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.544355 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86814002734d018b2052bdf9bdedab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.544939 5103 consensus_queue.cc:260] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.545044 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86814002734d018b2052bdf9bdedab (DEFAULT_TABLE table=default.Table_8de2edcca99e40cdaf5db0ea01f4a652 [id=baa0ab04a63642178ac0b350438c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.545913 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86814002734d018b2052bdf9bdedab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.548992 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 07a529884fc240da98aa910371610f3a (DEFAULT_TABLE table=default.Table_ebe4e5a3a5124836ab1d84342f373932 [id=b2089db97f8c461eaa1269b37ec8e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.551100 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfabcea41774bb9b922d5dd7acd881d (DEFAULT_TABLE table=default.Table_5171aa8dfe0c451c902f936b9587191d [id=c8105da569dd4f3a95e3fb2dbc419684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.551224 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfabcea41774bb9b922d5dd7acd881d (DEFAULT_TABLE table=default.Table_5171aa8dfe0c451c902f936b9587191d [id=c8105da569dd4f3a95e3fb2dbc419684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.551640 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfabcea41774bb9b922d5dd7acd881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.551963 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfabcea41774bb9b922d5dd7acd881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.554312 5106 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.554847 5106 ts_tablet_manager.cc:1367] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20240503 02:12:42.554910 5103 ts_tablet_manager.cc:1398] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:12:42.556725 5115 ts_tablet_manager.cc:1888] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.557264 5103 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.558625 5103 tablet_bootstrap.cc:654] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.557128 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011db39a7fa4b41903784d2c3f0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.559859 5115 log.cc:1196] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/1c2cdd9fdae843fe86c779e8eb67bfe2
I20240503 02:12:42.560658 4775 tablet_service.cc:1467] Processing CreateTablet for tablet bf091eb225ec4fa88264d438ac95dd59 (DEFAULT_TABLE table=default.Table_cc721583b82d4bc6b7a31189e77a3c17 [id=ddc85d902c66430baa162ef3db4886e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.561070 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf091eb225ec4fa88264d438ac95dd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.561120 5115 ts_tablet_manager.cc:1909] T 1c2cdd9fdae843fe86c779e8eb67bfe2 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.558975 6638 consensus_queue.cc:1035] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.559273 5106 raft_consensus.cc:348] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.567648 5106 raft_consensus.cc:374] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.567711 5106 raft_consensus.cc:729] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.567926 5106 consensus_queue.cc:260] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.568179 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 712) took 3975 ms. Trace:
I20240503 02:12:42.568261 5115 rpcz_store.cc:276] 0503 02:12:38.585739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.585775 (+ 36us) service_pool.cc:225] Handling call
0503 02:12:42.495090 (+3909315us) ts_tablet_manager.cc:1120] Deleting tablet 1c2cdd9fdae843fe86c779e8eb67bfe2
0503 02:12:42.495096 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.532534 (+ 37438us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.556623 (+ 24089us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.561409 (+ 4786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3909168}
I20240503 02:12:42.567360 4494 tablet_service.cc:1467] Processing CreateTablet for tablet bf091eb225ec4fa88264d438ac95dd59 (DEFAULT_TABLE table=default.Table_cc721583b82d4bc6b7a31189e77a3c17 [id=ddc85d902c66430baa162ef3db4886e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.568817 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf091eb225ec4fa88264d438ac95dd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.569077 5106 ts_tablet_manager.cc:1398] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 02:12:42.575361 5115 tablet_replica.cc:330] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.575592 5115 raft_consensus.cc:2229] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.577301 5106 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.578914 5106 tablet_bootstrap.cc:654] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.580675 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a529884fc240da98aa910371610f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.582564 6620 raft_consensus.cc:482] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.582705 6620 raft_consensus.cc:504] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.583278 6620 leader_election.cc:288] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.584254 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.584879 4462 leader_election.cc:341] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.585229 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.585531 4811 raft_consensus.cc:2454] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:42.585920 4462 leader_election.cc:302] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:42.586365 6620 raft_consensus.cc:2781] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.586457 6620 raft_consensus.cc:482] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.586512 6620 raft_consensus.cc:3037] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.589184 6620 raft_consensus.cc:504] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.589826 6620 leader_election.cc:288] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.590790 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.590967 4806 raft_consensus.cc:3037] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.593658 4806 raft_consensus.cc:2454] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.594206 4462 leader_election.cc:302] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.594655 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
W20240503 02:12:42.595242 4462 leader_election.cc:341] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.595634 6620 raft_consensus.cc:2781] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.595896 6620 raft_consensus.cc:686] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:42.596189 6620 consensus_queue.cc:237] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.599869 5103 tablet_bootstrap.cc:492] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.600093 5103 ts_tablet_manager.cc:1367] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20240503 02:12:42.601267 5103 raft_consensus.cc:348] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.601521 5103 raft_consensus.cc:374] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.601621 5103 raft_consensus.cc:729] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.601874 5103 consensus_queue.cc:260] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.602486 5103 ts_tablet_manager.cc:1398] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:42.602752 5103 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.604215 5103 tablet_bootstrap.cc:654] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.618708 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9e4dabd6e05948b98f02515023e2aaa1
I20240503 02:12:42.619691 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_72c760ead308416095fdc381a2737426" } new_table_name: "default.Table_c8853925301e4c99909b380247e366bc" modify_external_catalogs: true
I20240503 02:12:42.619868 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 2152f205cf1b4ec7ae03605bbfa007e1 (table default.Table_2b80e65a3283458d8556b0832d236304 [id=5bcf2302fc054796817b56de0e6e1eb8]) successfully deleted
I20240503 02:12:42.620404 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86814002734d018b2052bdf9bdedab (DEFAULT_TABLE table=default.Table_8de2edcca99e40cdaf5db0ea01f4a652 [id=baa0ab04a63642178ac0b350438c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.620852 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86814002734d018b2052bdf9bdedab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.627240 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.632776 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfabcea41774bb9b922d5dd7acd881d (DEFAULT_TABLE table=default.Table_5171aa8dfe0c451c902f936b9587191d [id=c8105da569dd4f3a95e3fb2dbc419684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.633412 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfabcea41774bb9b922d5dd7acd881d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.658241 6595 raft_consensus.cc:482] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.658438 6595 raft_consensus.cc:504] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.659159 6595 leader_election.cc:288] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.660167 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:42.660830 4746 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.663651 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.664294 4746 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.668538 4746 leader_election.cc:302] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.669025 6595 raft_consensus.cc:2726] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:42.700856 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 81037.4us
W20240503 02:12:42.701366 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3e6a3ca705e8482a8cd6a6fb17d93470 because the tablet was not found. No further retry: Not found: Tablet not found: 3e6a3ca705e8482a8cd6a6fb17d93470
W20240503 02:12:42.701792 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c because the tablet was not found. No further retry: Not found: Tablet not found: 11b5f5d5bd4a4fd6a0cfcfb8cc9e924c
I20240503 02:12:42.702343 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 1c2cdd9fdae843fe86c779e8eb67bfe2 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
I20240503 02:12:42.704147 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:42.705596 5105 tablet_bootstrap.cc:492] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:42.707582 5115 raft_consensus.cc:2258] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.708544 5115 ts_tablet_manager.cc:1875] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.712357 6317 raft_consensus.cc:482] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.712534 6317 raft_consensus.cc:504] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.713148 6317 leader_election.cc:288] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.715314 6262 consensus_queue.cc:1035] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:42.716384 4608 consensus_peers.cc:485] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7461aa76a1af41cb9253285bda708f30. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:42.717070 5105 ts_tablet_manager.cc:1367] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.244s user 0.003s sys 0.001s
I20240503 02:12:42.717875 5105 raft_consensus.cc:348] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.718650 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035216c9475d4b59aa1ac81b285b4d8a" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.718809 5105 raft_consensus.cc:374] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.718956 5105 raft_consensus.cc:729] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.719266 5105 consensus_queue.cc:260] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.725339 5105 ts_tablet_manager.cc:1398] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:42.726555 5105 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:42.734117 6595 raft_consensus.cc:482] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.734548 6595 raft_consensus.cc:504] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.735267 6595 leader_election.cc:288] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.736171 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1436dd3df7284f9bb379236eb9d76836" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:42.736162 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1436dd3df7284f9bb379236eb9d76836" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:42.736866 4746 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:42.737133 4746 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.737255 4746 leader_election.cc:302] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.737715 6595 raft_consensus.cc:2726] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.741425 5115 ts_tablet_manager.cc:1888] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.741921 5115 log.cc:1196] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/53017352b8f34955b116bbfd32f04854
I20240503 02:12:42.742359 5115 ts_tablet_manager.cc:1909] T 53017352b8f34955b116bbfd32f04854 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.754804 6616 raft_consensus.cc:482] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.755051 6616 raft_consensus.cc:504] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.755738 6616 leader_election.cc:288] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.756472 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43119471f91c4a1ba900f267c88db9a6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:42.756758 4675 raft_consensus.cc:2454] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:42.757359 4462 leader_election.cc:302] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.757699 6616 raft_consensus.cc:2781] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.757809 6616 raft_consensus.cc:482] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.757918 6616 raft_consensus.cc:3037] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:42.761775 4462 leader_election.cc:341] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 43119471f91c4a1ba900f267c88db9a6
I20240503 02:12:42.760833 6616 raft_consensus.cc:504] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.763271 6616 leader_election.cc:288] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.766736 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43119471f91c4a1ba900f267c88db9a6" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:42.766991 4675 raft_consensus.cc:3037] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.769877 4675 raft_consensus.cc:2454] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.771445 4462 leader_election.cc:302] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:42.774715 4462 leader_election.cc:341] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 43119471f91c4a1ba900f267c88db9a6
I20240503 02:12:42.775136 6620 raft_consensus.cc:2781] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.775310 6620 raft_consensus.cc:686] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:42.775614 6620 consensus_queue.cc:237] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.778381 5103 tablet_bootstrap.cc:492] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:42.778584 5103 ts_tablet_manager.cc:1367] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.176s user 0.003s sys 0.000s
I20240503 02:12:42.779417 5103 raft_consensus.cc:348] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.779712 5103 raft_consensus.cc:374] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.779819 5103 raft_consensus.cc:729] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.780086 5103 consensus_queue.cc:260] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.780659 5103 ts_tablet_manager.cc:1398] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:42.781116 5103 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:42.782721 5103 tablet_bootstrap.cc:654] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.788307 4648 tablet_service.cc:1467] Processing CreateTablet for tablet bf091eb225ec4fa88264d438ac95dd59 (DEFAULT_TABLE table=default.Table_cc721583b82d4bc6b7a31189e77a3c17 [id=ddc85d902c66430baa162ef3db4886e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.723572 5159 ts_tablet_manager.cc:1888] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:42.788766 5159 log.cc:1196] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/76b01907833d44f6a000a4de3a2a6e96
I20240503 02:12:42.789187 5159 ts_tablet_manager.cc:1909] T 76b01907833d44f6a000a4de3a2a6e96 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:42.790040 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 814) took 2778 ms. Trace:
W20240503 02:12:42.802884 4608 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:42.805034 4608 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:42.805240 4608 leader_election.cc:302] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.805516 6638 raft_consensus.cc:2726] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.805987 6638 consensus_queue.cc:1035] T 556f8938f35f42bfbbfb27f829653d45 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.809641 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b2560f50415444b8b2e497453a558d3a
I20240503 02:12:42.810128 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b5f444b22112407ca02fe86ac50167a9
I20240503 02:12:42.810539 4269 catalog_manager.cc:6554] ReplaceTablet: tablet b5f444b22112407ca02fe86ac50167a9 deleted and replaced by tablet d6caafb7adff4f91a6a5756769eebde0
I20240503 02:12:42.810726 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet a6de1b96c15c4b44809a03ef99e077ba
I20240503 02:12:42.823352 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 713) took 4237 ms. Trace:
I20240503 02:12:42.823482 5115 rpcz_store.cc:276] 0503 02:12:38.586069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.586115 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:42.568527 (+3982412us) ts_tablet_manager.cc:1120] Deleting tablet 53017352b8f34955b116bbfd32f04854
0503 02:12:42.575273 (+ 6746us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.725515 (+150242us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.741299 (+ 15784us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.823325 (+ 82026us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3982281}
I20240503 02:12:42.823776 5115 tablet_replica.cc:330] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.823948 5115 raft_consensus.cc:2229] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:42.824306 5115 raft_consensus.cc:2258] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.824868 5115 ts_tablet_manager.cc:1875] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.832242 5115 ts_tablet_manager.cc:1888] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.832466 5115 log.cc:1196] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/f2f9b962c9264f87b798df33544cb15d
I20240503 02:12:42.832859 5115 ts_tablet_manager.cc:1909] T f2f9b962c9264f87b798df33544cb15d P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:42.836778 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 714) took 4250 ms. Trace:
I20240503 02:12:42.836982 5115 rpcz_store.cc:276] 0503 02:12:38.586392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.586434 (+ 42us) service_pool.cc:225] Handling call
0503 02:12:42.823731 (+4237297us) ts_tablet_manager.cc:1120] Deleting tablet f2f9b962c9264f87b798df33544cb15d
0503 02:12:42.823740 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.827925 (+ 4185us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.832131 (+ 4206us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.836759 (+ 4628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4237169}
I20240503 02:12:42.837440 5115 tablet_replica.cc:330] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:42.837662 5115 raft_consensus.cc:2229] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:42.844805 5115 raft_consensus.cc:2258] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.845153 5159 rpcz_store.cc:276] 0503 02:12:40.011500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.011549 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:42.306056 (+2294507us) ts_tablet_manager.cc:1120] Deleting tablet 76b01907833d44f6a000a4de3a2a6e96
0503 02:12:42.306067 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.533861 (+227794us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.723394 (+189533us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.790020 (+ 66626us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294356}
I20240503 02:12:42.819612 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:42.837944 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf091eb225ec4fa88264d438ac95dd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.847165 5106 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.850317 5106 ts_tablet_manager.cc:1367] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.273s user 0.081s sys 0.116s
I20240503 02:12:42.851086 5106 raft_consensus.cc:348] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.851298 5106 raft_consensus.cc:374] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.851346 5106 raft_consensus.cc:729] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.851552 5106 consensus_queue.cc:260] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.852018 5106 ts_tablet_manager.cc:1398] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:42.852288 5106 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.854368 5106 tablet_bootstrap.cc:654] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.861696 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Bootstrap replaying log segment 1/1 (220.9K/258.4K this segment, stats: ops{read=589 overwritten=0 applied=588 ignored=0} inserts{seen=537 ignored=0} mutations{seen=670 ignored=0} orphaned_commits=0)
I20240503 02:12:42.864066 5159 tablet_replica.cc:330] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:42.864321 5159 raft_consensus.cc:2229] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.864504 5159 raft_consensus.cc:2258] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.865162 5159 ts_tablet_manager.cc:1875] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.868983 5115 ts_tablet_manager.cc:1875] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.874992 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 2931b4ef0f9c4041aad0f50be21d4ac4 (DEFAULT_TABLE table=default.Table_1829205c0cf64fd3ab99c62e91ab9229 [id=02a3b7b78da54b678f3300a06dae6bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.877458 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2931b4ef0f9c4041aad0f50be21d4ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.884625 6595 raft_consensus.cc:482] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.884790 6595 raft_consensus.cc:504] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.885478 6595 leader_election.cc:288] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.886430 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043ba14eaa5b46319b0b88457d2a5f45" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.886965 4746 leader_election.cc:341] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.790513 5114 ts_tablet_manager.cc:1888] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:42.888223 5114 log.cc:1196] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/41d44fe8f6334572996665bdec60e2e5
I20240503 02:12:42.888615 5114 ts_tablet_manager.cc:1909] T 41d44fe8f6334572996665bdec60e2e5 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:42.888978 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 754) took 2999 ms. Trace:
I20240503 02:12:42.889071 5114 rpcz_store.cc:276] 0503 02:12:39.889442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.889913 (+ 471us) service_pool.cc:225] Handling call
0503 02:12:42.463785 (+2573872us) ts_tablet_manager.cc:1120] Deleting tablet 41d44fe8f6334572996665bdec60e2e5
0503 02:12:42.463793 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.474549 (+ 10756us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.790391 (+315842us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:42.888955 (+ 98564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2573731}
I20240503 02:12:42.889652 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 758) took 2999 ms. Trace:
I20240503 02:12:42.889736 5114 rpcz_store.cc:276] 0503 02:12:39.889877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.890130 (+ 253us) service_pool.cc:225] Handling call
0503 02:12:42.889518 (+2999388us) ts_tablet_manager.cc:1120] Deleting tablet 44707b6439b141b7b63f49f34f25b37a
0503 02:12:42.889527 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:42.889636 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2999223}
I20240503 02:12:42.835292 6620 raft_consensus.cc:482] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.873713 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 2931b4ef0f9c4041aad0f50be21d4ac4 (DEFAULT_TABLE table=default.Table_1829205c0cf64fd3ab99c62e91ab9229 [id=02a3b7b78da54b678f3300a06dae6bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.874975 4491 tablet_service.cc:1467] Processing CreateTablet for tablet ef1370d66cb2420eaafd51dc64d43455 (DEFAULT_TABLE table=default.Table_0b12b36e91cd4d519c2e1277d3d97637 [id=106bf33ac8754d509ad86027159ab7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.886137 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043ba14eaa5b46319b0b88457d2a5f45" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.890337 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2931b4ef0f9c4041aad0f50be21d4ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.893225 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1370d66cb2420eaafd51dc64d43455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.897559 5105 tablet_bootstrap.cc:654] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:42.898332 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 192460518d49417cb8f03ac6ec971eef (table default.Table_c24f2a34ca8a43a388947bcc6a82f54e [id=739fc0c312224e5a8c60ee6d817b1879], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 192460518d49417cb8f03ac6ec971eef consensus state has no leader
I20240503 02:12:42.899013 5106 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.899201 5106 ts_tablet_manager.cc:1367] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.001s
I20240503 02:12:42.900215 5106 raft_consensus.cc:348] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.900442 5106 raft_consensus.cc:374] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.900539 5106 raft_consensus.cc:729] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.900782 5106 consensus_queue.cc:260] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.901475 5106 ts_tablet_manager.cc:1398] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:42.901785 5106 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:42.903168 4746 leader_election.cc:341] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.903383 5106 tablet_bootstrap.cc:654] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.903378 4746 leader_election.cc:302] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.903833 6595 raft_consensus.cc:482] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.903970 6595 raft_consensus.cc:504] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.904575 6595 leader_election.cc:288] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.904700 6595 raft_consensus.cc:2726] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.905015 6620 raft_consensus.cc:504] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.905686 6620 leader_election.cc:288] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.906494 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.907058 4462 leader_election.cc:341] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.908689 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:42.908859 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:42.909087 4811 raft_consensus.cc:2454] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:42.909408 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.910058 4746 leader_election.cc:341] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:42.910284 4746 leader_election.cc:341] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.910392 4746 leader_election.cc:302] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.910724 6661 raft_consensus.cc:2726] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.910843 4462 leader_election.cc:302] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81
I20240503 02:12:42.911823 6620 raft_consensus.cc:2781] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.911942 6620 raft_consensus.cc:482] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.912053 6620 raft_consensus.cc:3037] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.914746 6620 raft_consensus.cc:504] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.915674 6620 leader_election.cc:288] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:42.916508 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:42.917336 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0494ee4bd746069bfaf39bfb46687d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:42.917589 4811 raft_consensus.cc:3037] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.921051 4811 raft_consensus.cc:2454] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:42.921633 4462 leader_election.cc:302] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:42.921867 5105 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:42.922003 4462 leader_election.cc:341] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.922472 5114 tablet_replica.cc:330] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:42.922703 5114 raft_consensus.cc:2229] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:42.922904 5114 raft_consensus.cc:2258] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:42.923485 5114 ts_tablet_manager.cc:1875] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:42.927354 5106 tablet_bootstrap.cc:492] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.927783 5106 ts_tablet_manager.cc:1367] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240503 02:12:42.929167 5106 raft_consensus.cc:348] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.929543 5106 raft_consensus.cc:374] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.929661 5106 raft_consensus.cc:729] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.930008 5106 consensus_queue.cc:260] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.931203 5106 ts_tablet_manager.cc:1398] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:42.932240 5106 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.933063 5105 ts_tablet_manager.cc:1367] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.207s user 0.022s sys 0.002s
I20240503 02:12:42.928779 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:42.933279 6616 raft_consensus.cc:2781] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.933724 6616 raft_consensus.cc:686] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:42.933802 5105 raft_consensus.cc:348] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.933988 5105 raft_consensus.cc:374] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.934036 5105 raft_consensus.cc:729] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.933990 6616 consensus_queue.cc:237] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.934232 5105 consensus_queue.cc:260] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.934875 5105 ts_tablet_manager.cc:1398] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:42.925720 4783 tablet_service.cc:1467] Processing CreateTablet for tablet ef1370d66cb2420eaafd51dc64d43455 (DEFAULT_TABLE table=default.Table_0b12b36e91cd4d519c2e1277d3d97637 [id=106bf33ac8754d509ad86027159ab7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.935199 5105 tablet_bootstrap.cc:492] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:42.935648 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1370d66cb2420eaafd51dc64d43455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.936627 5105 tablet_bootstrap.cc:654] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.937760 6661 raft_consensus.cc:482] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.937886 6661 raft_consensus.cc:504] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.938516 6661 leader_election.cc:288] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.939206 5106 tablet_bootstrap.cc:654] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.939949 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:42.940670 4746 leader_election.cc:341] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.939868 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.942018 4746 leader_election.cc:341] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.942183 4746 leader_election.cc:302] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.942456 6661 raft_consensus.cc:2726] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.949465 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8a99467cf14ac194fa684730109308 (DEFAULT_TABLE table=default.Table_56cb6b9e2faa4ecb8a243d55becff404 [id=38c7caa8a25142b980459ff2f375f8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.949992 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8a99467cf14ac194fa684730109308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.953063 6661 raft_consensus.cc:482] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.953225 6661 raft_consensus.cc:504] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.953889 6661 leader_election.cc:288] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:42.954677 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:42.955543 4746 leader_election.cc:341] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.955852 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:42.960325 4746 leader_election.cc:341] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:42.960522 4746 leader_election.cc:302] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:42.960829 6661 raft_consensus.cc:2726] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:42.962085 5106 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.962432 5106 ts_tablet_manager.cc:1367] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20240503 02:12:42.964102 5106 raft_consensus.cc:348] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.964694 5106 raft_consensus.cc:374] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.964905 5106 raft_consensus.cc:729] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.965382 5106 consensus_queue.cc:260] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.966531 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8a99467cf14ac194fa684730109308 (DEFAULT_TABLE table=default.Table_56cb6b9e2faa4ecb8a243d55becff404 [id=38c7caa8a25142b980459ff2f375f8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.966867 4494 tablet_service.cc:1467] Processing CreateTablet for tablet fabf6535ce7a4df4a3445547a8e91acb (DEFAULT_TABLE table=default.Table_206d524a9f2542b9847dc17ed534ae92 [id=e57b5d0272144922936894f1dd978298]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.967075 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8a99467cf14ac194fa684730109308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.967232 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf6535ce7a4df4a3445547a8e91acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.967530 5106 ts_tablet_manager.cc:1398] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:42.968647 5106 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.969480 4780 tablet_service.cc:1467] Processing CreateTablet for tablet fabf6535ce7a4df4a3445547a8e91acb (DEFAULT_TABLE table=default.Table_206d524a9f2542b9847dc17ed534ae92 [id=e57b5d0272144922936894f1dd978298]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.969885 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf6535ce7a4df4a3445547a8e91acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.974998 5106 tablet_bootstrap.cc:654] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.987097 5106 tablet_bootstrap.cc:492] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:42.987437 5106 ts_tablet_manager.cc:1367] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20240503 02:12:42.989367 5106 raft_consensus.cc:348] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.989576 5106 raft_consensus.cc:374] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.989624 5106 raft_consensus.cc:729] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.989791 5106 consensus_queue.cc:260] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:42.990357 5106 ts_tablet_manager.cc:1398] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:42.990617 5106 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:42.992306 5106 tablet_bootstrap.cc:654] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.003994 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b2560f50415444b8b2e497453a558d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.003679 5103 tablet_bootstrap.cc:492] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.004017 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet b5f444b22112407ca02fe86ac50167a9 with delete_type TABLET_DATA_DELETED (replaced by tablet d6caafb7adff4f91a6a5756769eebde0) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.004798 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet a6de1b96c15c4b44809a03ef99e077ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.005120 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b5f444b22112407ca02fe86ac50167a9 with delete_type TABLET_DATA_DELETED (replaced by tablet d6caafb7adff4f91a6a5756769eebde0) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.005439 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 76b01907833d44f6a000a4de3a2a6e96 (table default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]) successfully deleted
I20240503 02:12:43.005928 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 53017352b8f34955b116bbfd32f04854 (table default.Table_bb18c51dac8c46b2a5ddecca6ddd34be [id=47856b6b36f74b5589f9acc60162c1e8]) successfully deleted
I20240503 02:12:43.006078 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet f2f9b962c9264f87b798df33544cb15d (table default.Table_bc9150faaae244e8bc1953025cc8c19f [id=dbc52c7d9081493e88c2f707ac450f5f]) successfully deleted
I20240503 02:12:43.006897 4507 tablet_service.cc:1467] Processing CreateTablet for tablet eebf7adb21f440e2b63561693646771b (DEFAULT_TABLE table=default.Table_866c8697e32646dc860d3cbc3856e41e [id=288bbdd4661b46998a59552fe99f00d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.007025 4783 tablet_service.cc:1467] Processing CreateTablet for tablet eebf7adb21f440e2b63561693646771b (DEFAULT_TABLE table=default.Table_866c8697e32646dc860d3cbc3856e41e [id=288bbdd4661b46998a59552fe99f00d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.007320 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebf7adb21f440e2b63561693646771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.007422 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebf7adb21f440e2b63561693646771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.007490 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 6caae07d88af48988d5619efcca508f5 (DEFAULT_TABLE table=default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5 [id=86e52fc3c66949fbbe7693ead3ba62aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.007836 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caae07d88af48988d5619efcca508f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.009094 4497 tablet_service.cc:1467] Processing CreateTablet for tablet a70506e39cd742ed81ac6c7133ae5095 (DEFAULT_TABLE table=default.Table_82219dec0a7945719fe7415d05e92a13 [id=a57f5f79650d48eca0921b73e03635e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.009407 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70506e39cd742ed81ac6c7133ae5095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.013577 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet a6de1b96c15c4b44809a03ef99e077ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.022162 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 192460518d49417cb8f03ac6ec971eef
I20240503 02:12:43.022264 4775 tablet_service.cc:1467] Processing CreateTablet for tablet a70506e39cd742ed81ac6c7133ae5095 (DEFAULT_TABLE table=default.Table_82219dec0a7945719fe7415d05e92a13 [id=a57f5f79650d48eca0921b73e03635e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.022706 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70506e39cd742ed81ac6c7133ae5095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.022852 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet aa67857b858c4719b498291c52b7aceb from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.022890 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_278fc530c46a46159b5cff1d43aa3a5b" } new_table_name: "default.Table_d3a9d71ceb774275880045dde6532bec" modify_external_catalogs: true
I20240503 02:12:43.023043 6661 raft_consensus.cc:482] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.023169 6661 raft_consensus.cc:504] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.023483 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9df571d7b21a4cce8f3f5423c9d50a37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.023707 6661 leader_election.cc:288] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:43.024106 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9df571d7b21a4cce8f3f5423c9d50a37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.025233 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.025403 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0f61ff50b4434dbbaae6066ce10e5f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.025992 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f61ff50b4434dbbaae6066ce10e5f6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.026227 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:43.026710 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2b00f84bf3e64c22b77142696ebcf830" } new_table_name: "default.Table_aabd4d0faa1c4125805cfae21e766300" modify_external_catalogs: true
I20240503 02:12:43.028034 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 2931b4ef0f9c4041aad0f50be21d4ac4 (DEFAULT_TABLE table=default.Table_1829205c0cf64fd3ab99c62e91ab9229 [id=02a3b7b78da54b678f3300a06dae6bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.028429 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2931b4ef0f9c4041aad0f50be21d4ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.029564 4269 catalog_manager.cc:5484] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:43.029996 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:43.029923 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4b0e3fb2ecb24f7ea287a2cbabe12de8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.030351 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b0e3fb2ecb24f7ea287a2cbabe12de8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.030227 4269 catalog_manager.cc:5484] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:43.030581 4269 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:43.031590 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_84073519a2f04b50863dc1ea8bac73b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.032292 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_41d45c372f894144a4e54926d63a14b8" } new_table_name: "default.Table_ef0fd51ad2644a6da603a6120e1203aa" modify_external_catalogs: true
I20240503 02:12:43.035316 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f10b8b1d68874a60a7900ec6fcc6e78c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.035733 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f10b8b1d68874a60a7900ec6fcc6e78c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:43.039083 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84073519a2f04b50863dc1ea8bac73b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.041445 4645 tablet_service.cc:1467] Processing CreateTablet for tablet ef1370d66cb2420eaafd51dc64d43455 (DEFAULT_TABLE table=default.Table_0b12b36e91cd4d519c2e1277d3d97637 [id=106bf33ac8754d509ad86027159ab7c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.042004 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8a99467cf14ac194fa684730109308 (DEFAULT_TABLE table=default.Table_56cb6b9e2faa4ecb8a243d55becff404 [id=38c7caa8a25142b980459ff2f375f8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.042062 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet b5f444b22112407ca02fe86ac50167a9 with delete_type TABLET_DATA_DELETED (replaced by tablet d6caafb7adff4f91a6a5756769eebde0) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.042366 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet a6de1b96c15c4b44809a03ef99e077ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.042447 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet b2560f50415444b8b2e497453a558d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.042749 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.042855 4646 tablet_service.cc:1467] Processing CreateTablet for tablet eebf7adb21f440e2b63561693646771b (DEFAULT_TABLE table=default.Table_866c8697e32646dc860d3cbc3856e41e [id=288bbdd4661b46998a59552fe99f00d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.042376 4647 tablet_service.cc:1467] Processing CreateTablet for tablet fabf6535ce7a4df4a3445547a8e91acb (DEFAULT_TABLE table=default.Table_206d524a9f2542b9847dc17ed534ae92 [id=e57b5d0272144922936894f1dd978298]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:43.043215 4608 consensus_peers.cc:485] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c9090b11c8a4bf0b4b92523b9431b79. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.043314 4638 tablet_service.cc:1467] Processing CreateTablet for tablet a70506e39cd742ed81ac6c7133ae5095 (DEFAULT_TABLE table=default.Table_82219dec0a7945719fe7415d05e92a13 [id=a57f5f79650d48eca0921b73e03635e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:43.043681 4746 leader_election.cc:341] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:43.044344 4746 leader_election.cc:341] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.044503 4746 leader_election.cc:302] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.044754 6661 raft_consensus.cc:2726] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:43.043359 4608 consensus_peers.cc:485] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a90ebcf29004d1a925b69fa20998dd3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.047147 5106 tablet_bootstrap.cc:492] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.047292 5106 ts_tablet_manager.cc:1367] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20240503 02:12:43.047972 5106 raft_consensus.cc:348] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.048220 5106 raft_consensus.cc:374] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.048316 5106 raft_consensus.cc:729] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.048563 5106 consensus_queue.cc:260] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.049149 5103 ts_tablet_manager.cc:1367] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.268s user 0.004s sys 0.000s
W20240503 02:12:43.051240 6294 log.cc:924] Time spent T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Append to log took a long time: real 0.055s user 0.003s sys 0.000s
I20240503 02:12:43.052124 5103 raft_consensus.cc:348] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.052904 5103 raft_consensus.cc:374] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.053254 5103 raft_consensus.cc:729] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.054641 5103 consensus_queue.cc:260] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.056777 5114 ts_tablet_manager.cc:1888] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.064726 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 6caae07d88af48988d5619efcca508f5 (DEFAULT_TABLE table=default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5 [id=86e52fc3c66949fbbe7693ead3ba62aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.067166 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caae07d88af48988d5619efcca508f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:43.050304 4608 consensus_peers.cc:485] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9511e4de6c942b9b9e0790c88d3d543. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.069435 5106 ts_tablet_manager.cc:1398] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:12:43.070308 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 6e779878cdff45deb809f9e2ba393629 (DEFAULT_TABLE table=default.Table_f124b2c9e27d43678750918ac22d43c1 [id=2930765932d94901be542d9a7d3b6553]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.070787 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e779878cdff45deb809f9e2ba393629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.075423 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=693 overwritten=0 applied=693 ignored=0} inserts{seen=639 ignored=0} mutations{seen=807 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:43.076931 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: Bootstrap complete.
I20240503 02:12:43.072024 4775 tablet_service.cc:1467] Processing CreateTablet for tablet ef64cff211664efab3dc41a1f26c3de1 (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.076207 5105 tablet_bootstrap.cc:492] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.082742 5105 ts_tablet_manager.cc:1367] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.148s user 0.021s sys 0.025s
I20240503 02:12:43.082819 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef64cff211664efab3dc41a1f26c3de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.082860 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1370d66cb2420eaafd51dc64d43455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.083437 5105 raft_consensus.cc:348] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.083616 5105 raft_consensus.cc:374] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.083662 5105 raft_consensus.cc:729] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.083813 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 41d44fe8f6334572996665bdec60e2e5 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:43.083838 5105 consensus_queue.cc:260] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:43.084020 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 44707b6439b141b7b63f49f34f25b37a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44707b6439b141b7b63f49f34f25b37a already in progress: creating tablet
I20240503 02:12:43.084360 5105 ts_tablet_manager.cc:1398] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:43.084635 5105 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.050995 6262 raft_consensus.cc:482] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.085132 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 192460518d49417cb8f03ac6ec971eef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.085238 6262 raft_consensus.cc:504] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.085636 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.085860 6262 leader_election.cc:288] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.069867 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 6caae07d88af48988d5619efcca508f5 (DEFAULT_TABLE table=default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5 [id=86e52fc3c66949fbbe7693ead3ba62aa]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:43.070169 4608 consensus_peers.cc:485] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c9090b11c8a4bf0b4b92523b9431b79. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:43.086611 4608 consensus_peers.cc:485] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:43.086743 4608 consensus_peers.cc:485] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.086967 4783 tablet_service.cc:1467] Processing CreateTablet for tablet ab0f1a936f8846d6b9b43de6d6fa3efe (DEFAULT_TABLE table=default.Table_e0dab12ce8a54de18f9e00962d9a1c6a [id=88ebede1428248d7be06ab71efd47545]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.087333 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0f1a936f8846d6b9b43de6d6fa3efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.087397 6638 consensus_queue.cc:1035] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.087381 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4dabd6e05948b98f02515023e2aaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.089027 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.089354 4806 raft_consensus.cc:2379] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:43.089984 5105 tablet_bootstrap.cc:654] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.091584 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8a99467cf14ac194fa684730109308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.093132 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebf7adb21f440e2b63561693646771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.097205 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70506e39cd742ed81ac6c7133ae5095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.100800 5105 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.101042 5105 ts_tablet_manager.cc:1367] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240503 02:12:43.101192 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf6535ce7a4df4a3445547a8e91acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.105759 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caae07d88af48988d5619efcca508f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.108052 5105 raft_consensus.cc:348] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.109463 5105 raft_consensus.cc:374] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.110227 5105 raft_consensus.cc:729] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.111759 5105 consensus_queue.cc:260] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.073921 5115 ts_tablet_manager.cc:1888] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.115478 5115 log.cc:1196] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7394c9d374404fb199ab6ff170d7febe
I20240503 02:12:43.116461 5105 ts_tablet_manager.cc:1398] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:12:43.118937 5115 ts_tablet_manager.cc:1909] T 7394c9d374404fb199ab6ff170d7febe P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.118826 5105 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.121482 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 715) took 4534 ms. Trace:
I20240503 02:12:43.122001 5115 rpcz_store.cc:276] 0503 02:12:38.586716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:38.586761 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:42.837385 (+4250624us) ts_tablet_manager.cc:1120] Deleting tablet 7394c9d374404fb199ab6ff170d7febe
0503 02:12:42.837394 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.050644 (+213250us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.073817 (+ 23173us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.121300 (+ 47483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4250460}
I20240503 02:12:43.125249 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 726) took 3972 ms. Trace:
I20240503 02:12:43.126112 5115 rpcz_store.cc:276] 0503 02:12:39.152688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.152751 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:43.124236 (+3971485us) ts_tablet_manager.cc:1120] Deleting tablet a4a16052d43d45aeb405c5de5dafa041
0503 02:12:43.124246 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.125232 (+ 986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3971083}
I20240503 02:12:43.128769 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 727) took 3939 ms. Trace:
I20240503 02:12:43.130373 5106 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.131683 5105 tablet_bootstrap.cc:654] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.132045 5106 tablet_bootstrap.cc:654] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:43.137724 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b94c407d27984998a550bef0d6f48c33 consensus state has no leader
I20240503 02:12:43.140877 4273 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet efa7c0c5fc614c16985008b8fba54bab (table default.Table_4d9a0ac169f94dfbad1496820ded5f07 [id=475bf67c3074494eb43e612a6897f57b]). Expected version 1 got 0
I20240503 02:12:43.144373 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_07f00cb83ef24520a8b4e944ab24b09f" } modify_external_catalogs: true
I20240503 02:12:43.144824 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_07f00cb83ef24520a8b4e944ab24b09f" } modify_external_catalogs: true
I20240503 02:12:43.149787 6294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:43.151551 5105 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.151858 5105 ts_tablet_manager.cc:1367] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240503 02:12:43.153043 6294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145afaa7548f4b7fafb23f20276f0cb4, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.153422 5105 raft_consensus.cc:348] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.153728 5105 raft_consensus.cc:374] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.153959 5159 ts_tablet_manager.cc:1888] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.154103 5159 log.cc:1196] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/66ca08877cd948e1b04736b54bc779db
I20240503 02:12:43.154385 5159 ts_tablet_manager.cc:1909] T 66ca08877cd948e1b04736b54bc779db P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.155097 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 806) took 3147 ms. Trace:
I20240503 02:12:43.155193 5159 rpcz_store.cc:276] 0503 02:12:40.007886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.019846 (+ 11960us) service_pool.cc:225] Handling call
0503 02:12:42.849755 (+2829909us) ts_tablet_manager.cc:1120] Deleting tablet 66ca08877cd948e1b04736b54bc779db
0503 02:12:42.863991 (+ 14236us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.038861 (+174870us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.153861 (+115000us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.155074 (+ 1213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824043}
I20240503 02:12:43.153955 5105 raft_consensus.cc:729] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.155751 6294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 693, Last appended: 5.693, Last appended by leader: 693, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:43.155825 5105 consensus_queue.cc:260] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.156306 5105 ts_tablet_manager.cc:1398] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:43.156662 5105 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.158079 5105 tablet_bootstrap.cc:654] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:43.159946 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 73906.1us
I20240503 02:12:43.160663 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 66ca08877cd948e1b04736b54bc779db (table default.Table_e8ca78c1991543479153a973a8346732 [id=997231149d3e427f824ed751b345d7b8]) successfully deleted
I20240503 02:12:43.056897 5114 log.cc:1196] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/a74f4816cad04a9eb5934aaba86c2fbc
I20240503 02:12:43.058063 5103 ts_tablet_manager.cc:1398] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:43.162900 5103 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.164618 5103 tablet_bootstrap.cc:654] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.072014 4497 tablet_service.cc:1467] Processing CreateTablet for tablet ef64cff211664efab3dc41a1f26c3de1 (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.165360 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef64cff211664efab3dc41a1f26c3de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.165776 5114 ts_tablet_manager.cc:1909] T a74f4816cad04a9eb5934aaba86c2fbc P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.166518 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 757) took 3276 ms. Trace:
I20240503 02:12:43.166615 5114 rpcz_store.cc:276] 0503 02:12:39.889758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.890681 (+ 923us) service_pool.cc:225] Handling call
0503 02:12:42.922411 (+3031730us) ts_tablet_manager.cc:1120] Deleting tablet a74f4816cad04a9eb5934aaba86c2fbc
0503 02:12:42.922425 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.053882 (+131457us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.056678 (+ 2796us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.166496 (+109818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3031592}
I20240503 02:12:43.167009 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 759) took 3277 ms. Trace:
I20240503 02:12:43.167089 5114 rpcz_store.cc:276] 0503 02:12:39.890031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.890106 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:43.166890 (+3276784us) ts_tablet_manager.cc:1120] Deleting tablet ab093285c39b4ce7a0bb3de9d016a918
0503 02:12:43.166899 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.166994 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3274659}
I20240503 02:12:43.167336 5114 tablet_replica.cc:330] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.167518 5114 raft_consensus.cc:2229] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.167661 5114 raft_consensus.cc:2258] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.168139 5114 ts_tablet_manager.cc:1875] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.077795 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 6e779878cdff45deb809f9e2ba393629 (DEFAULT_TABLE table=default.Table_f124b2c9e27d43678750918ac22d43c1 [id=2930765932d94901be542d9a7d3b6553]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.170714 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e779878cdff45deb809f9e2ba393629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.170730 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:43.089062 4507 tablet_service.cc:1467] Processing CreateTablet for tablet ab0f1a936f8846d6b9b43de6d6fa3efe (DEFAULT_TABLE table=default.Table_e0dab12ce8a54de18f9e00962d9a1c6a [id=88ebede1428248d7be06ab71efd47545]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.171838 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0f1a936f8846d6b9b43de6d6fa3efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.129449 5115 rpcz_store.cc:276] 0503 02:12:39.189067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.189131 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:43.127974 (+3938843us) ts_tablet_manager.cc:1120] Deleting tablet 81d6d3ca80c54c629a5de82b06a7aac4
0503 02:12:43.127987 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.128750 (+ 763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3938467}
I20240503 02:12:43.172305 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 728) took 3983 ms. Trace:
I20240503 02:12:43.172382 5115 rpcz_store.cc:276] 0503 02:12:39.189188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.190265 (+ 1077us) service_pool.cc:225] Handling call
0503 02:12:43.172175 (+3981910us) ts_tablet_manager.cc:1120] Deleting tablet 16b3d1e7ce83425db0bd56457d9f2544
0503 02:12:43.172185 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.172287 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981719}
I20240503 02:12:43.172696 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 729) took 3982 ms. Trace:
I20240503 02:12:43.091109 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 192460518d49417cb8f03ac6ec971eef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.172755 5115 rpcz_store.cc:276] 0503 02:12:39.190621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.190961 (+ 340us) service_pool.cc:225] Handling call
0503 02:12:43.172592 (+3981631us) ts_tablet_manager.cc:1120] Deleting tablet a3374e095846463a859bb642c2f42167
0503 02:12:43.172599 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.172684 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981487}
I20240503 02:12:43.091959 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91798b4bc2f74b9289425531937a78ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.173008 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 730) took 3958 ms. Trace:
I20240503 02:12:43.173064 5115 rpcz_store.cc:276] 0503 02:12:39.214429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.214483 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:43.172907 (+3958424us) ts_tablet_manager.cc:1120] Deleting tablet fc67bba49712445db6d784b6ed94dd7e
0503 02:12:43.172935 (+ 28us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.172996 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958242}
I20240503 02:12:43.114078 6620 consensus_queue.cc:1035] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20240503 02:12:43.173291 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 735) took 3785 ms. Trace:
I20240503 02:12:43.176491 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet a74f4816cad04a9eb5934aaba86c2fbc (table default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]) successfully deleted
W20240503 02:12:43.176761 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ab093285c39b4ce7a0bb3de9d016a918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab093285c39b4ce7a0bb3de9d016a918 already in progress: creating tablet
I20240503 02:12:43.177162 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 825) took 2804 ms. Trace:
I20240503 02:12:43.177258 5159 rpcz_store.cc:276] 0503 02:12:40.372596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.372650 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:43.177033 (+2804383us) ts_tablet_manager.cc:1120] Deleting tablet 73198ec769a9486da3f85e8fcbef8e78
0503 02:12:43.177044 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.177142 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804156}
I20240503 02:12:43.177608 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 826) took 2804 ms. Trace:
I20240503 02:12:43.177675 5159 rpcz_store.cc:276] 0503 02:12:40.373159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.373205 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:43.177505 (+2804300us) ts_tablet_manager.cc:1120] Deleting tablet 13e31219b2134fa080572238ff0be424
0503 02:12:43.177513 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.177591 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804095}
I20240503 02:12:43.177966 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 827) took 2779 ms. Trace:
I20240503 02:12:43.178035 5159 rpcz_store.cc:276] 0503 02:12:40.398843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.399374 (+ 531us) service_pool.cc:225] Handling call
0503 02:12:43.177869 (+2778495us) ts_tablet_manager.cc:1120] Deleting tablet a8ecddfcfed2436e98bb7b6b38beb03f
0503 02:12:43.177879 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.177949 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2778314}
I20240503 02:12:43.178349 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 828) took 2750 ms. Trace:
I20240503 02:12:43.178411 5159 rpcz_store.cc:276] 0503 02:12:40.428251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.428310 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:43.178253 (+2749943us) ts_tablet_manager.cc:1120] Deleting tablet 3b92999890ee49d78d7c16f5ed1c4958
0503 02:12:43.178261 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.178332 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749756}
I20240503 02:12:43.178699 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 829) took 2748 ms. Trace:
I20240503 02:12:43.178763 5159 rpcz_store.cc:276] 0503 02:12:40.430514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.430743 (+ 229us) service_pool.cc:225] Handling call
0503 02:12:43.178607 (+2747864us) ts_tablet_manager.cc:1120] Deleting tablet fca2c34e0fc44be8b7ee9252ef557497
0503 02:12:43.178615 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.178684 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2747659}
I20240503 02:12:43.179014 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 830) took 2729 ms. Trace:
I20240503 02:12:43.179080 5159 rpcz_store.cc:276] 0503 02:12:40.449416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.449479 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:43.178933 (+2729454us) ts_tablet_manager.cc:1120] Deleting tablet c919bda1eaaa4e52828aac3ef337f3e5
0503 02:12:43.178941 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.179000 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2729252}
I20240503 02:12:43.179373 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 832) took 2488 ms. Trace:
I20240503 02:12:43.179442 5159 rpcz_store.cc:276] 0503 02:12:40.690837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.690910 (+ 73us) service_pool.cc:225] Handling call
0503 02:12:43.179287 (+2488377us) ts_tablet_manager.cc:1120] Deleting tablet 0ab2b01fb13c4aaa931b3a633ab5d847
0503 02:12:43.179295 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.179359 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30,"tablet-delete.queue_time_us":2488168}
I20240503 02:12:43.179440 6619 consensus_queue.cc:1035] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.179772 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 831) took 2491 ms. Trace:
I20240503 02:12:43.179845 5159 rpcz_store.cc:276] 0503 02:12:40.688150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.688736 (+ 586us) service_pool.cc:225] Handling call
0503 02:12:43.179681 (+2490945us) ts_tablet_manager.cc:1120] Deleting tablet 07225ae9ad3c4e44ac0c5c51421b0115
0503 02:12:43.179690 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.179759 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2488207}
I20240503 02:12:43.180132 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 833) took 2462 ms. Trace:
I20240503 02:12:43.180194 5105 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.180338 5105 ts_tablet_manager.cc:1367] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240503 02:12:43.181031 5105 raft_consensus.cc:348] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.181234 5105 raft_consensus.cc:374] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.181296 5105 raft_consensus.cc:729] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.181515 5105 consensus_queue.cc:260] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.182008 5105 ts_tablet_manager.cc:1398] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.182274 5105 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.135337 6616 raft_consensus.cc:482] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.182840 6616 raft_consensus.cc:504] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.183614 6616 leader_election.cc:288] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.183764 5105 tablet_bootstrap.cc:654] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.180217 5159 rpcz_store.cc:276] 0503 02:12:40.717495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.717866 (+ 371us) service_pool.cc:225] Handling call
0503 02:12:43.180037 (+2462171us) ts_tablet_manager.cc:1120] Deleting tablet 676d5f52102d496081df3964722b286f
0503 02:12:43.180047 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.180117 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461905}
I20240503 02:12:43.184981 5159 tablet_replica.cc:330] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.185155 5159 raft_consensus.cc:2229] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.185566 5159 raft_consensus.cc:2258] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.186053 5159 ts_tablet_manager.cc:1875] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:43.188297 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75aa4b8ef74e4af18aa4390d593d665d consensus state has no leader
I20240503 02:12:43.191282 6294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:43.192178 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_15995c9366f647308e1abed005edeb79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.188859 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
W20240503 02:12:43.192987 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15995c9366f647308e1abed005edeb79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.193100 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:12:43.194659 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e5ddaa3ca6c4d499538b9900ca8c6a2 consensus state has no leader
I20240503 02:12:43.194988 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_fb49809c3ca04cf9ac1cad7770bd1897" } new_table_name: "default.Table_64cb0d91e91245c3a93537f8b7d4f778" modify_external_catalogs: true
I20240503 02:12:43.195359 5114 ts_tablet_manager.cc:1888] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.195470 5114 log.cc:1196] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/78b21a3010a4470288fbdd684ef64e77
I20240503 02:12:43.195765 5114 ts_tablet_manager.cc:1909] T 78b21a3010a4470288fbdd684ef64e77 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.196523 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 151f925dcb304594b60c85d850b83533 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.196518 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 761) took 3301 ms. Trace:
I20240503 02:12:43.196600 5114 rpcz_store.cc:276] 0503 02:12:39.895458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.895861 (+ 403us) service_pool.cc:225] Handling call
0503 02:12:43.167297 (+3271436us) ts_tablet_manager.cc:1120] Deleting tablet 78b21a3010a4470288fbdd684ef64e77
0503 02:12:43.167305 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.186549 (+ 19244us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.195239 (+ 8690us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.196499 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3271323}
I20240503 02:12:43.196934 5114 tablet_replica.cc:330] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.197014 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet cbaa94834cdc44eabe71242a698e3639 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.197116 5114 raft_consensus.cc:2229] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.197768 5105 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.197969 5105 ts_tablet_manager.cc:1367] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:12:43.198638 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a7c7733ae249477b9cbd82203830f7a6" } modify_external_catalogs: true
I20240503 02:12:43.198634 5105 raft_consensus.cc:348] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.198819 5105 raft_consensus.cc:374] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.198869 5105 raft_consensus.cc:729] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.198827 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a7c7733ae249477b9cbd82203830f7a6" } modify_external_catalogs: true
I20240503 02:12:43.199047 5105 consensus_queue.cc:260] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.199589 5105 ts_tablet_manager.cc:1398] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.199856 5105 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.200526 5114 raft_consensus.cc:2258] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.200842 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet f8be9e5380b4466baccd16483e940588 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.201156 5114 ts_tablet_manager.cc:1875] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.204730 5159 ts_tablet_manager.cc:1888] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.204838 5159 log.cc:1196] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/78e9faaa6a584ec38b9da9d2670e47f3
I20240503 02:12:43.205302 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:43.205432 4275 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1df47adfa50a41229b5672b7c3d1794b consensus state has no leader
I20240503 02:12:43.205528 4806 raft_consensus.cc:2454] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:43.205797 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b70b33ab1fd34982a6f9463002af9bac" } modify_external_catalogs: true
I20240503 02:12:43.205935 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b70b33ab1fd34982a6f9463002af9bac" } modify_external_catalogs: true
I20240503 02:12:43.206116 5105 tablet_bootstrap.cc:654] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.206528 4462 leader_election.cc:302] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.063896 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet b2560f50415444b8b2e497453a558d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.212806 6616 raft_consensus.cc:2781] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.212970 6616 raft_consensus.cc:482] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.213048 6616 raft_consensus.cc:3037] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.214171 5159 ts_tablet_manager.cc:1909] T 78e9faaa6a584ec38b9da9d2670e47f3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:43.184545 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 97435.3us
I20240503 02:12:43.202749 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:43.215065 4667 raft_consensus.cc:2454] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:43.216282 4647 tablet_service.cc:1467] Processing CreateTablet for tablet ef64cff211664efab3dc41a1f26c3de1 (DEFAULT_TABLE table=default.Table_cd19aee3b1c2424a8295717b4227f6d3 [id=e35ba6c5729d4b5e864ab2f90bf68d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.216814 6317 raft_consensus.cc:482] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.216853 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef64cff211664efab3dc41a1f26c3de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.216888 4608 leader_election.cc:302] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.216980 6317 raft_consensus.cc:504] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.217486 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 6e779878cdff45deb809f9e2ba393629 (DEFAULT_TABLE table=default.Table_f124b2c9e27d43678750918ac22d43c1 [id=2930765932d94901be542d9a7d3b6553]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.217557 6317 leader_election.cc:288] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.217847 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e779878cdff45deb809f9e2ba393629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.217952 6317 raft_consensus.cc:3037] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.220499 6317 raft_consensus.cc:2726] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.221462 4644 tablet_service.cc:1467] Processing CreateTablet for tablet ab0f1a936f8846d6b9b43de6d6fa3efe (DEFAULT_TABLE table=default.Table_e0dab12ce8a54de18f9e00962d9a1c6a [id=88ebede1428248d7be06ab71efd47545]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.221860 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0f1a936f8846d6b9b43de6d6fa3efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.223251 4528 raft_consensus.cc:3037] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.225023 6262 raft_consensus.cc:482] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.225142 6262 raft_consensus.cc:504] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.225804 6262 leader_election.cc:288] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.225941 6262 raft_consensus.cc:482] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.226075 6262 raft_consensus.cc:504] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.226579 6262 leader_election.cc:288] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.228790 6616 raft_consensus.cc:504] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.229977 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:43.230226 4667 raft_consensus.cc:3037] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.232601 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 192460518d49417cb8f03ac6ec971eef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:42 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.234891 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.235020 4608 consensus_peers.cc:485] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76b01907833d44f6a000a4de3a2a6e96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.235050 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38d63d2cc8c4f54ba43cdf8cde1841f" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.235188 4528 raft_consensus.cc:2454] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.236316 4667 raft_consensus.cc:2454] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:43.236809 4462 leader_election.cc:302] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.237149 6619 raft_consensus.cc:2781] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.237246 6619 raft_consensus.cc:686] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:43.237481 6619 consensus_queue.cc:237] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.240428 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.240785 6638 consensus_queue.cc:1035] T de4e96b4be094101bf74904b2d66b6c7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:43.241596 4608 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.241570 6317 consensus_queue.cc:1035] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.241765 4608 leader_election.cc:302] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.214918 5115 rpcz_store.cc:276] 0503 02:12:39.387876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.387939 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:43.173220 (+3785281us) ts_tablet_manager.cc:1120] Deleting tablet 868a6ad6cc1a40f091b4516676c24c97
0503 02:12:43.173226 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.173281 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3785081}
I20240503 02:12:43.242434 5115 tablet_replica.cc:330] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:43.242677 5115 raft_consensus.cc:2229] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.242735 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.242854 5115 raft_consensus.cc:2258] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.243005 4808 raft_consensus.cc:2454] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.243619 5115 ts_tablet_manager.cc:1875] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.244269 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.244467 4806 raft_consensus.cc:2454] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.244486 6560 consensus_queue.cc:1035] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:43.245186 4608 leader_election.cc:341] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.247192 4608 leader_election.cc:302] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:43.247704 4608 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:43.247870 6638 consensus_queue.cc:1035] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.248391 6638 raft_consensus.cc:2781] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.248476 6638 raft_consensus.cc:482] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.248560 6638 raft_consensus.cc:3037] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.251653 6638 raft_consensus.cc:504] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.252259 6638 leader_election.cc:288] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.245332 6262 raft_consensus.cc:2781] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.252810 6262 raft_consensus.cc:482] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.252907 6262 raft_consensus.cc:3037] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.254367 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7394c9d374404fb199ab6ff170d7febe (table default.Table_be53cf9a5c3a418d9dc24120c17d3a4b [id=646347328e89442b8713fa69f5078052]) successfully deleted
W20240503 02:12:43.254715 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a4a16052d43d45aeb405c5de5dafa041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4a16052d43d45aeb405c5de5dafa041 already in progress: creating tablet
I20240503 02:12:43.254711 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:43.254902 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 81d6d3ca80c54c629a5de82b06a7aac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81d6d3ca80c54c629a5de82b06a7aac4 already in progress: creating tablet
W20240503 02:12:43.255173 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a3374e095846463a859bb642c2f42167 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3374e095846463a859bb642c2f42167 already in progress: creating tablet
W20240503 02:12:43.255385 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet fc67bba49712445db6d784b6ed94dd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc67bba49712445db6d784b6ed94dd7e already in progress: creating tablet
I20240503 02:12:43.256035 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 78b21a3010a4470288fbdd684ef64e77 (table default.Table_7566b876f57547599a98c4f55fbfc427 [id=5d52bb117988409d974ea961f9f7bab9]) successfully deleted
W20240503 02:12:43.256515 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 73198ec769a9486da3f85e8fcbef8e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73198ec769a9486da3f85e8fcbef8e78 already in progress: creating tablet
W20240503 02:12:43.256780 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 13e31219b2134fa080572238ff0be424 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13e31219b2134fa080572238ff0be424 already in progress: creating tablet
W20240503 02:12:43.257025 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a8ecddfcfed2436e98bb7b6b38beb03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ecddfcfed2436e98bb7b6b38beb03f already in progress: creating tablet
W20240503 02:12:43.257337 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3b92999890ee49d78d7c16f5ed1c4958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b92999890ee49d78d7c16f5ed1c4958 already in progress: creating tablet
W20240503 02:12:43.257561 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet fca2c34e0fc44be8b7ee9252ef557497 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca2c34e0fc44be8b7ee9252ef557497 already in progress: creating tablet
I20240503 02:12:43.258483 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:43.255318 6262 raft_consensus.cc:504] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.262336 6595 raft_consensus.cc:482] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.262557 6595 raft_consensus.cc:504] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.262842 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.263451 6595 leader_election.cc:288] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.263526 6262 leader_election.cc:288] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.264505 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.265683 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:43.266422 4746 leader_election.cc:341] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:43.266705 4746 leader_election.cc:341] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.266798 4746 leader_election.cc:302] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.267359 6595 raft_consensus.cc:2726] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:43.267771 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet c919bda1eaaa4e52828aac3ef337f3e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c919bda1eaaa4e52828aac3ef337f3e5 already in progress: creating tablet
W20240503 02:12:43.268273 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab2b01fb13c4aaa931b3a633ab5d847 already in progress: creating tablet
W20240503 02:12:43.269852 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 07225ae9ad3c4e44ac0c5c51421b0115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07225ae9ad3c4e44ac0c5c51421b0115 already in progress: creating tablet
I20240503 02:12:43.269896 6616 leader_election.cc:288] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.270761 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f862891b43a4845b723bfdb9a7da591" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.270964 5114 ts_tablet_manager.cc:1888] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.271005 4813 raft_consensus.cc:3037] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.271101 5114 log.cc:1196] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/95681e67db1e4485876d530133e8a514
W20240503 02:12:43.272536 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 676d5f52102d496081df3964722b286f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676d5f52102d496081df3964722b286f already in progress: creating tablet
I20240503 02:12:43.273573 5115 ts_tablet_manager.cc:1888] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.271593 5114 ts_tablet_manager.cc:1909] T 95681e67db1e4485876d530133e8a514 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.273974 5115 log.cc:1196] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/4225c1bc133d4e18a4a47fd58820ca10
I20240503 02:12:43.274516 4813 raft_consensus.cc:2454] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:43.275035 5115 ts_tablet_manager.cc:1909] T 4225c1bc133d4e18a4a47fd58820ca10 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.275590 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 736) took 3886 ms. Trace:
I20240503 02:12:43.274523 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 762) took 3378 ms. Trace:
I20240503 02:12:43.276073 5114 rpcz_store.cc:276] 0503 02:12:39.895587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.895745 (+ 158us) service_pool.cc:225] Handling call
0503 02:12:43.196869 (+3301124us) ts_tablet_manager.cc:1120] Deleting tablet 95681e67db1e4485876d530133e8a514
0503 02:12:43.196877 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.205246 (+ 8369us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.270812 (+ 65566us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.274489 (+ 3677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3300799}
I20240503 02:12:43.276892 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 95681e67db1e4485876d530133e8a514 (table default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]) successfully deleted
I20240503 02:12:43.247891 4608 leader_election.cc:302] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.277426 5115 rpcz_store.cc:276] 0503 02:12:39.388834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.388888 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:43.242384 (+3853496us) ts_tablet_manager.cc:1120] Deleting tablet 4225c1bc133d4e18a4a47fd58820ca10
0503 02:12:43.242394 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.250932 (+ 8538us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.273387 (+ 22455us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.275562 (+ 2175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3853305}
I20240503 02:12:43.278761 5115 tablet_replica.cc:330] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:43.278326 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 763) took 3382 ms. Trace:
I20240503 02:12:43.278961 5114 rpcz_store.cc:276] 0503 02:12:39.895697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.896090 (+ 393us) service_pool.cc:225] Handling call
0503 02:12:43.277836 (+3381746us) ts_tablet_manager.cc:1120] Deleting tablet 4fb59bbf7b68431c86d88f27ef2528b0
0503 02:12:43.278128 (+ 292us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.278279 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3381528}
I20240503 02:12:43.279506 5115 raft_consensus.cc:2229] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.279474 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.279716 4813 raft_consensus.cc:3037] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.281075 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66f88a24b74f2c9b2050af081d4601" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:43.282944 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4fb59bbf7b68431c86d88f27ef2528b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fb59bbf7b68431c86d88f27ef2528b0 already in progress: creating tablet
I20240503 02:12:43.283404 5114 tablet_replica.cc:330] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.279795 6638 raft_consensus.cc:2726] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.284035 5115 pending_rounds.cc:62] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:43.284050 4813 raft_consensus.cc:2454] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.284299 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.284523 4808 raft_consensus.cc:3037] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.284646 5114 raft_consensus.cc:2229] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.285164 5114 raft_consensus.cc:2258] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.285900 5114 ts_tablet_manager.cc:1875] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.287706 4808 raft_consensus.cc:2454] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.287731 5115 pending_rounds.cc:68] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420873854427136 op_type: NO_OP noop_request { }
I20240503 02:12:43.287928 5115 raft_consensus.cc:2866] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:43.288074 5115 raft_consensus.cc:2258] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.289414 4608 leader_election.cc:302] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:43.289810 6638 raft_consensus.cc:2781] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.289947 6638 raft_consensus.cc:686] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.290267 5115 ts_tablet_manager.cc:1875] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.290362 6638 consensus_queue.cc:237] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.292245 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b173d78ba74ede801b12cccbeef6b1" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:43.292455 4528 raft_consensus.cc:3037] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.294991 4528 raft_consensus.cc:2454] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.297518 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 4225c1bc133d4e18a4a47fd58820ca10 (table default.Table_bb63ba232a074c0692fa657db43e25dc [id=5cbfe61d82174908b3b00ff5d287eb8f]) successfully deleted
I20240503 02:12:43.301415 4608 leader_election.cc:302] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:43.301930 4608 leader_election.cc:341] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.302467 6262 raft_consensus.cc:2781] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.302650 6262 raft_consensus.cc:686] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.303337 6262 consensus_queue.cc:237] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.308614 5114 ts_tablet_manager.cc:1888] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.309453 5115 ts_tablet_manager.cc:1888] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.309679 5114 log.cc:1196] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:43.310114 5114 ts_tablet_manager.cc:1909] T ea7afb429d64497abcd9f1d0910651c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.310495 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 756) took 3420 ms. Trace:
I20240503 02:12:43.310689 5115 log.cc:1196] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/6c9090b11c8a4bf0b4b92523b9431b79
I20240503 02:12:43.311125 5114 rpcz_store.cc:276] 0503 02:12:39.889673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.897267 (+ 7594us) service_pool.cc:225] Handling call
0503 02:12:43.283344 (+3386077us) ts_tablet_manager.cc:1120] Deleting tablet ea7afb429d64497abcd9f1d0910651c3
0503 02:12:43.283359 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.294363 (+ 11004us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.308487 (+ 14124us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.310470 (+ 1983us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3385899}
I20240503 02:12:43.311601 5115 ts_tablet_manager.cc:1909] T 6c9090b11c8a4bf0b4b92523b9431b79 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.312002 5114 tablet_replica.cc:330] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.312274 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet ea7afb429d64497abcd9f1d0910651c3 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
I20240503 02:12:43.312381 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 737) took 3907 ms. Trace:
I20240503 02:12:43.312640 5114 raft_consensus.cc:2229] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.312968 5115 rpcz_store.cc:276] 0503 02:12:39.405062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.410089 (+ 5027us) service_pool.cc:225] Handling call
0503 02:12:43.278693 (+3868604us) ts_tablet_manager.cc:1120] Deleting tablet 6c9090b11c8a4bf0b4b92523b9431b79
0503 02:12:43.278707 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.298389 (+ 19682us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.309344 (+ 10955us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.312362 (+ 3018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3868313}
I20240503 02:12:43.312875 5114 raft_consensus.cc:2258] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.313884 5114 ts_tablet_manager.cc:1875] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.314354 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 743) took 3842 ms. Trace:
I20240503 02:12:43.314502 5115 rpcz_store.cc:276] 0503 02:12:39.471944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.471997 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:43.314203 (+3842206us) ts_tablet_manager.cc:1120] Deleting tablet 1716c4807e004760b34c0492cb7f522d
0503 02:12:43.314217 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.314336 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3842029}
I20240503 02:12:43.314915 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 745) took 3778 ms. Trace:
I20240503 02:12:43.316025 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 6c9090b11c8a4bf0b4b92523b9431b79 (table default.Table_e6fc3292e398448cbb24c71e0c98ec56 [id=5283ee29bf5142f0928e9aa919fb924a]) successfully deleted
I20240503 02:12:43.316397 5115 rpcz_store.cc:276] 0503 02:12:39.536631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.536689 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:43.314807 (+3778118us) ts_tablet_manager.cc:1120] Deleting tablet f20e573d70bd46aab33466ccfc801465
0503 02:12:43.314815 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.314902 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3777894}
W20240503 02:12:43.316449 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1716c4807e004760b34c0492cb7f522d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1716c4807e004760b34c0492cb7f522d already in progress: creating tablet
I20240503 02:12:43.317010 5115 tablet_replica.cc:330] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:43.317664 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f20e573d70bd46aab33466ccfc801465 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20e573d70bd46aab33466ccfc801465 already in progress: creating tablet
I20240503 02:12:43.323791 5114 ts_tablet_manager.cc:1888] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.324041 5114 log.cc:1196] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/f146f7cce0b24d3087bfe0973e2b7ef0
I20240503 02:12:43.324458 5114 ts_tablet_manager.cc:1909] T f146f7cce0b24d3087bfe0973e2b7ef0 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.324827 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 760) took 3429 ms. Trace:
I20240503 02:12:43.325037 5114 rpcz_store.cc:276] 0503 02:12:39.895325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.901438 (+ 6113us) service_pool.cc:225] Handling call
0503 02:12:43.311953 (+3410515us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:43.311963 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.320004 (+ 8041us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.323358 (+ 3354us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.324805 (+ 1447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3410306}
I20240503 02:12:43.325985 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet f146f7cce0b24d3087bfe0973e2b7ef0 (table default.Table_18dc31f244e94d6ab3c576eaae5e408b [id=64f4026243214e1bbed391c6e0e7df76]) successfully deleted
I20240503 02:12:43.326723 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 764) took 3378 ms. Trace:
I20240503 02:12:43.326890 5114 rpcz_store.cc:276] 0503 02:12:39.947824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.948482 (+ 658us) service_pool.cc:225] Handling call
0503 02:12:43.326471 (+3377989us) ts_tablet_manager.cc:1120] Deleting tablet 41531ddcbf66465094d8f06e1116b3be
0503 02:12:43.326488 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.326696 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3377765}
W20240503 02:12:43.327819 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 41531ddcbf66465094d8f06e1116b3be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41531ddcbf66465094d8f06e1116b3be already in progress: creating tablet
I20240503 02:12:43.328320 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 765) took 3376 ms. Trace:
I20240503 02:12:43.328450 5114 rpcz_store.cc:276] 0503 02:12:39.951919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.952004 (+ 85us) service_pool.cc:225] Handling call
0503 02:12:43.328197 (+3376193us) ts_tablet_manager.cc:1120] Deleting tablet 95c7e4a2954e4161b9acc029873f923b
0503 02:12:43.328208 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.328303 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3376017}
W20240503 02:12:43.328992 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 95c7e4a2954e4161b9acc029873f923b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c7e4a2954e4161b9acc029873f923b already in progress: creating tablet
I20240503 02:12:43.329452 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 781) took 2958 ms. Trace:
I20240503 02:12:43.329566 5114 rpcz_store.cc:276] 0503 02:12:40.370903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.371227 (+ 324us) service_pool.cc:225] Handling call
0503 02:12:43.329312 (+2958085us) ts_tablet_manager.cc:1120] Deleting tablet 13e31219b2134fa080572238ff0be424
0503 02:12:43.329322 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.329433 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2957890}
W20240503 02:12:43.330114 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 13e31219b2134fa080572238ff0be424 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13e31219b2134fa080572238ff0be424 already in progress: creating tablet
I20240503 02:12:43.330528 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 782) took 2959 ms. Trace:
I20240503 02:12:43.330639 5114 rpcz_store.cc:276] 0503 02:12:40.371045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.371450 (+ 405us) service_pool.cc:225] Handling call
0503 02:12:43.330422 (+2958972us) ts_tablet_manager.cc:1120] Deleting tablet 73198ec769a9486da3f85e8fcbef8e78
0503 02:12:43.330432 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.330512 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2958730}
W20240503 02:12:43.331082 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 73198ec769a9486da3f85e8fcbef8e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73198ec769a9486da3f85e8fcbef8e78 already in progress: creating tablet
I20240503 02:12:43.331455 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 783) took 2932 ms. Trace:
I20240503 02:12:43.331565 5114 rpcz_store.cc:276] 0503 02:12:40.399194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.399848 (+ 654us) service_pool.cc:225] Handling call
0503 02:12:43.331345 (+2931497us) ts_tablet_manager.cc:1120] Deleting tablet a8ecddfcfed2436e98bb7b6b38beb03f
0503 02:12:43.331354 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.331439 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2931286}
W20240503 02:12:43.332279 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a8ecddfcfed2436e98bb7b6b38beb03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ecddfcfed2436e98bb7b6b38beb03f already in progress: creating tablet
I20240503 02:12:43.332772 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 785) took 2904 ms. Trace:
I20240503 02:12:43.332882 5114 rpcz_store.cc:276] 0503 02:12:40.428338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.428546 (+ 208us) service_pool.cc:225] Handling call
0503 02:12:43.332673 (+2904127us) ts_tablet_manager.cc:1120] Deleting tablet fca2c34e0fc44be8b7ee9252ef557497
0503 02:12:43.332683 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.332756 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2903690}
W20240503 02:12:43.333384 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet fca2c34e0fc44be8b7ee9252ef557497 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca2c34e0fc44be8b7ee9252ef557497 already in progress: creating tablet
I20240503 02:12:43.333884 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 784) took 2906 ms. Trace:
I20240503 02:12:43.333999 5114 rpcz_store.cc:276] 0503 02:12:40.427214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.427712 (+ 498us) service_pool.cc:225] Handling call
0503 02:12:43.333768 (+2906056us) ts_tablet_manager.cc:1120] Deleting tablet 3b92999890ee49d78d7c16f5ed1c4958
0503 02:12:43.333777 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.333869 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2904075}
W20240503 02:12:43.334487 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3b92999890ee49d78d7c16f5ed1c4958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b92999890ee49d78d7c16f5ed1c4958 already in progress: creating tablet
I20240503 02:12:43.334856 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 786) took 2885 ms. Trace:
I20240503 02:12:43.334967 5114 rpcz_store.cc:276] 0503 02:12:40.449305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.451697 (+ 2392us) service_pool.cc:225] Handling call
0503 02:12:43.334761 (+2883064us) ts_tablet_manager.cc:1120] Deleting tablet c919bda1eaaa4e52828aac3ef337f3e5
0503 02:12:43.334770 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.334840 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2882366}
W20240503 02:12:43.335427 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet c919bda1eaaa4e52828aac3ef337f3e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c919bda1eaaa4e52828aac3ef337f3e5 already in progress: creating tablet
I20240503 02:12:43.335783 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 789) took 2647 ms. Trace:
I20240503 02:12:43.335888 5114 rpcz_store.cc:276] 0503 02:12:40.688409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.688838 (+ 429us) service_pool.cc:225] Handling call
0503 02:12:43.335692 (+2646854us) ts_tablet_manager.cc:1120] Deleting tablet 07225ae9ad3c4e44ac0c5c51421b0115
0503 02:12:43.335700 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.335769 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2611927}
W20240503 02:12:43.336342 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 07225ae9ad3c4e44ac0c5c51421b0115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07225ae9ad3c4e44ac0c5c51421b0115 already in progress: creating tablet
I20240503 02:12:43.336688 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 790) took 2646 ms. Trace:
I20240503 02:12:43.336791 5114 rpcz_store.cc:276] 0503 02:12:40.690586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.690651 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:43.336602 (+2645951us) ts_tablet_manager.cc:1120] Deleting tablet 0ab2b01fb13c4aaa931b3a633ab5d847
0503 02:12:43.336610 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.336675 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2612811}
W20240503 02:12:43.337235 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab2b01fb13c4aaa931b3a633ab5d847 already in progress: creating tablet
I20240503 02:12:43.337584 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 791) took 2639 ms. Trace:
I20240503 02:12:43.337699 5114 rpcz_store.cc:276] 0503 02:12:40.698188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.698249 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:43.337496 (+2639247us) ts_tablet_manager.cc:1120] Deleting tablet 676d5f52102d496081df3964722b286f
0503 02:12:43.337504 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.337570 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2613568}
W20240503 02:12:43.338158 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 676d5f52102d496081df3964722b286f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676d5f52102d496081df3964722b286f already in progress: creating tablet
I20240503 02:12:43.338472 5114 tablet_replica.cc:330] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.338685 5114 raft_consensus.cc:2229] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.339004 5114 raft_consensus.cc:2258] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.339617 5114 ts_tablet_manager.cc:1875] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.348351 5114 ts_tablet_manager.cc:1888] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.348618 5114 log.cc:1196] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/78e9faaa6a584ec38b9da9d2670e47f3
I20240503 02:12:43.349035 5114 ts_tablet_manager.cc:1909] T 78e9faaa6a584ec38b9da9d2670e47f3 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.349890 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 792) took 2644 ms. Trace:
I20240503 02:12:43.350030 5114 rpcz_store.cc:276] 0503 02:12:40.705245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.705309 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:43.338426 (+2633117us) ts_tablet_manager.cc:1120] Deleting tablet 78e9faaa6a584ec38b9da9d2670e47f3
0503 02:12:43.338435 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.343156 (+ 4721us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.348240 (+ 5084us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.349871 (+ 1631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2614131}
I20240503 02:12:43.350608 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 78e9faaa6a584ec38b9da9d2670e47f3 (table default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]) successfully deleted
I20240503 02:12:43.351089 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 796) took 2553 ms. Trace:
I20240503 02:12:43.351210 5114 rpcz_store.cc:276] 0503 02:12:40.797491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.797552 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:43.350954 (+2553402us) ts_tablet_manager.cc:1120] Deleting tablet b8287b45fa0b492ca82c22164f3a7837
0503 02:12:43.350964 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.351073 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553193}
W20240503 02:12:43.351758 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b8287b45fa0b492ca82c22164f3a7837 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8287b45fa0b492ca82c22164f3a7837 already in progress: creating tablet
I20240503 02:12:43.352172 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 804) took 2494 ms. Trace:
I20240503 02:12:43.352291 5114 rpcz_store.cc:276] 0503 02:12:40.857429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.857488 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:43.352065 (+2494577us) ts_tablet_manager.cc:1120] Deleting tablet a48d5eac44264899a7a372b9f85fb8f5
0503 02:12:43.352074 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.352157 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494359}
W20240503 02:12:43.352802 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a48d5eac44264899a7a372b9f85fb8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48d5eac44264899a7a372b9f85fb8f5 already in progress: creating tablet
I20240503 02:12:43.353003 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 834) took 2635 ms. Trace:
I20240503 02:12:43.353179 5114 tablet_replica.cc:330] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.353163 5159 rpcz_store.cc:276] 0503 02:12:40.717623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.717800 (+ 177us) service_pool.cc:225] Handling call
0503 02:12:43.184933 (+2467133us) ts_tablet_manager.cc:1120] Deleting tablet 78e9faaa6a584ec38b9da9d2670e47f3
0503 02:12:43.184943 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.201750 (+ 16807us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.204636 (+ 2886us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.352975 (+148339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466582}
I20240503 02:12:43.353746 5159 tablet_replica.cc:330] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.353691 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 78e9faaa6a584ec38b9da9d2670e47f3 (table default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]) successfully deleted
I20240503 02:12:43.353945 5159 raft_consensus.cc:2229] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.354101 5159 raft_consensus.cc:2258] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.354207 5114 raft_consensus.cc:2229] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.354482 5114 raft_consensus.cc:2258] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.355243 5114 ts_tablet_manager.cc:1875] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.363531 5114 ts_tablet_manager.cc:1888] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:43.369324 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65ba631d85854b43bcb4bece9bfba7d3 consensus state has no leader
I20240503 02:12:43.369874 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6175ee8a08c4522912024d62f15a984" } modify_external_catalogs: true
I20240503 02:12:43.370142 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e6175ee8a08c4522912024d62f15a984" } modify_external_catalogs: true
I20240503 02:12:43.371035 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet cbaa94834cdc44eabe71242a698e3639
I20240503 02:12:43.371646 4272 catalog_manager.cc:6554] ReplaceTablet: tablet cbaa94834cdc44eabe71242a698e3639 deleted and replaced by tablet b3c4ebfc21214cb9a5d863c927fe8776
I20240503 02:12:43.371958 4272 master_service.cc:946] ReplaceTablet: received request to replace tablet 04d555c8d8844ee49c960a5279485480 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.373203 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f8be9e5380b4466baccd16483e940588
I20240503 02:12:43.373824 4269 catalog_manager.cc:6554] ReplaceTablet: tablet f8be9e5380b4466baccd16483e940588 deleted and replaced by tablet aeb7b745bb994555a3e513f92ff29429
I20240503 02:12:43.374231 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.374965 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.379050 5103 tablet_bootstrap.cc:492] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.379141 5115 raft_consensus.cc:2229] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.379330 5103 ts_tablet_manager.cc:1367] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.003s
I20240503 02:12:43.379930 5115 pending_rounds.cc:62] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:43.379984 5115 pending_rounds.cc:68] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7023420880035741696 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "7461aa76a1af41cb9253285bda708f30" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_4f9e55262006461d8fd7dd706e640469" dest_uuid: "1f50719d94564844b15e1a9211d48a81" new_extra_config { } }
W20240503 02:12:43.380295 5115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57782: Aborted: Op aborted
I20240503 02:12:43.380735 5115 raft_consensus.cc:2258] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.381626 6560 consensus_queue.cc:1035] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.382277 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet cbaa94834cdc44eabe71242a698e3639 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c4ebfc21214cb9a5d863c927fe8776) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.382846 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet f8be9e5380b4466baccd16483e940588 with delete_type TABLET_DATA_DELETED (replaced by tablet aeb7b745bb994555a3e513f92ff29429) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.385541 6595 raft_consensus.cc:482] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.385695 6595 raft_consensus.cc:504] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.386415 6595 leader_election.cc:288] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.387089 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:43.388060 4746 leader_election.cc:341] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:43.392858 4608 consensus_peers.cc:485] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66ca08877cd948e1b04736b54bc779db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.391340 5159 ts_tablet_manager.cc:1875] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.398063 5103 raft_consensus.cc:348] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.398270 5103 raft_consensus.cc:374] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.372738 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2785fa72fa1e42629f30b99a8dc42925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.398324 5103 raft_consensus.cc:729] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.398607 5103 consensus_queue.cc:260] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:43.398839 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2785fa72fa1e42629f30b99a8dc42925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.399231 5103 ts_tablet_manager.cc:1398] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240503 02:12:43.399528 5103 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.399555 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet cbaa94834cdc44eabe71242a698e3639 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c4ebfc21214cb9a5d863c927fe8776) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.399780 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet f8be9e5380b4466baccd16483e940588 with delete_type TABLET_DATA_DELETED (replaced by tablet aeb7b745bb994555a3e513f92ff29429) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.404119 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet cbaa94834cdc44eabe71242a698e3639 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c4ebfc21214cb9a5d863c927fe8776) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.404421 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet f8be9e5380b4466baccd16483e940588 with delete_type TABLET_DATA_DELETED (replaced by tablet aeb7b745bb994555a3e513f92ff29429) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.404659 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.405040 6698 raft_consensus.cc:482] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:12:43.405267 4746 leader_election.cc:341] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.405256 6698 raft_consensus.cc:504] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.405443 4746 leader_election.cc:302] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.405848 6595 raft_consensus.cc:2726] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.405897 6698 leader_election.cc:288] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.407461 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.407779 4806 raft_consensus.cc:2379] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f50719d94564844b15e1a9211d48a81 in current term 1: Already voted for candidate f4f9f079230a468f90e5e3a58f254b91 in this term.
I20240503 02:12:43.409354 6699 consensus_queue.cc:1035] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.396133 5106 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.411448 5106 ts_tablet_manager.cc:1367] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.281s user 0.026s sys 0.069s
I20240503 02:12:43.411561 4275 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b3c4ebfc21214cb9a5d863c927fe8776
I20240503 02:12:43.412025 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2c4258a98cf14b69af1ffe992cf712e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.412276 4520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8e996d76be4dda876f4645df56c1ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.412636 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4258a98cf14b69af1ffe992cf712e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.417528 4608 leader_election.cc:302] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.398100 5115 ts_tablet_manager.cc:1875] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.418649 6698 raft_consensus.cc:3037] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.421241 6698 raft_consensus.cc:2726] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.422959 6595 raft_consensus.cc:482] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.423106 6595 raft_consensus.cc:504] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.423767 6595 leader_election.cc:288] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.423825 5105 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.424057 5105 ts_tablet_manager.cc:1367] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.224s user 0.005s sys 0.005s
I20240503 02:12:43.416101 5106 raft_consensus.cc:348] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.424363 5106 raft_consensus.cc:374] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.424475 5106 raft_consensus.cc:729] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.425720 5105 raft_consensus.cc:348] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.426074 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fc35d19d5cd54bc2824834be14a6bb14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.426501 5106 consensus_queue.cc:260] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:43.426710 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc35d19d5cd54bc2824834be14a6bb14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.429746 5106 ts_tablet_manager.cc:1398] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 02:12:43.429894 5105 raft_consensus.cc:374] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.430006 5105 raft_consensus.cc:729] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.430280 5105 consensus_queue.cc:260] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.430817 5106 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.431427 5103 tablet_bootstrap.cc:654] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.432446 5106 tablet_bootstrap.cc:654] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.433382 5105 ts_tablet_manager.cc:1398] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:43.429034 5159 ts_tablet_manager.cc:1888] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.433961 5159 log.cc:1196] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/cd66f88a24b74f2c9b2050af081d4601
I20240503 02:12:43.434484 5159 ts_tablet_manager.cc:1909] T cd66f88a24b74f2c9b2050af081d4601 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.434903 6699 raft_consensus.cc:482] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.435076 6699 raft_consensus.cc:504] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.445765 6699 leader_election.cc:288] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:43.447456 4608 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
W20240503 02:12:43.449090 4608 leader_election.cc:341] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:43.449281 4608 leader_election.cc:302] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.449685 6698 raft_consensus.cc:2726] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.451078 5115 ts_tablet_manager.cc:1888] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:43.454777 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.466477 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cab73492219d41c4bf3fafcf6ac31309" } modify_external_catalogs: true
I20240503 02:12:43.466959 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cab73492219d41c4bf3fafcf6ac31309" } modify_external_catalogs: true
I20240503 02:12:43.468192 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4ea17824609f4b25a85121fc6b203812" } new_table_name: "default.Table_512dd0dd609548978d52d05cc8e69782" modify_external_catalogs: true
I20240503 02:12:43.470538 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2e9fd306667548278723f880bb82a806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.471566 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e9fd306667548278723f880bb82a806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:43.473214 4746 leader_election.cc:341] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.454972 5106 tablet_bootstrap.cc:492] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.473649 5106 ts_tablet_manager.cc:1367] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20240503 02:12:43.474010 5105 tablet_bootstrap.cc:492] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.474433 5114 log.cc:1196] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3db94fb9d53044d796e2973143d888fb
I20240503 02:12:43.474364 5106 raft_consensus.cc:348] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.474570 5106 raft_consensus.cc:374] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.474619 5106 raft_consensus.cc:729] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.474838 5114 ts_tablet_manager.cc:1909] T 3db94fb9d53044d796e2973143d888fb P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.474843 5106 consensus_queue.cc:260] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.475155 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 812) took 2365 ms. Trace:
I20240503 02:12:43.475230 5114 rpcz_store.cc:276] 0503 02:12:41.109612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.109662 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:43.353119 (+2243457us) ts_tablet_manager.cc:1120] Deleting tablet 3db94fb9d53044d796e2973143d888fb
0503 02:12:43.353132 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.360170 (+ 7038us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.363421 (+ 3251us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.475123 (+111702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2205471}
I20240503 02:12:43.475364 5106 ts_tablet_manager.cc:1398] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.475625 5106 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.475723 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 3db94fb9d53044d796e2973143d888fb (table default.Table_3e7dad4cf88048efa489e61c2595ad65 [id=5ca70228a70e4c02995e3c058a569e68]) successfully deleted
I20240503 02:12:43.476056 5105 tablet_bootstrap.cc:654] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.476883 5106 tablet_bootstrap.cc:654] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.477675 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 835) took 2759 ms. Trace:
I20240503 02:12:43.477784 5159 rpcz_store.cc:276] 0503 02:12:40.717742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.717801 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:43.353691 (+2635890us) ts_tablet_manager.cc:1120] Deleting tablet cd66f88a24b74f2c9b2050af081d4601
0503 02:12:43.353701 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.396518 (+ 42817us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.428894 (+ 32376us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.477644 (+ 48750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635203}
I20240503 02:12:43.478312 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 840) took 2679 ms. Trace:
I20240503 02:12:43.478402 5159 rpcz_store.cc:276] 0503 02:12:40.798435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.798495 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:43.478165 (+2679670us) ts_tablet_manager.cc:1120] Deleting tablet b8287b45fa0b492ca82c22164f3a7837
0503 02:12:43.478175 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.478293 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2679484}
I20240503 02:12:43.478776 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 847) took 2620 ms. Trace:
I20240503 02:12:43.478863 5159 rpcz_store.cc:276] 0503 02:12:40.858248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.858302 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:43.478665 (+2620363us) ts_tablet_manager.cc:1120] Deleting tablet a48d5eac44264899a7a372b9f85fb8f5
0503 02:12:43.478675 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.478761 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620178}
I20240503 02:12:43.479348 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet cd66f88a24b74f2c9b2050af081d4601 (table default.Table_3d65228f6c2b4694bb60f3118ad65af7 [id=2d5bfb965c844a199fa81636be8265a6]) successfully deleted
W20240503 02:12:43.479723 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b8287b45fa0b492ca82c22164f3a7837 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8287b45fa0b492ca82c22164f3a7837 already in progress: creating tablet
I20240503 02:12:43.479823 5159 tablet_replica.cc:330] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: stopping tablet replica
W20240503 02:12:43.480000 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a48d5eac44264899a7a372b9f85fb8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48d5eac44264899a7a372b9f85fb8f5 already in progress: creating tablet
I20240503 02:12:43.480094 5159 raft_consensus.cc:2229] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.480285 5159 raft_consensus.cc:2258] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.481178 5159 ts_tablet_manager.cc:1875] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.486550 5115 log.cc:1196] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/7461aa76a1af41cb9253285bda708f30
I20240503 02:12:43.487012 5115 ts_tablet_manager.cc:1909] T 7461aa76a1af41cb9253285bda708f30 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.487371 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 750) took 3847 ms. Trace:
I20240503 02:12:43.488018 5115 rpcz_store.cc:276] 0503 02:12:39.639976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.643764 (+ 3788us) service_pool.cc:225] Handling call
0503 02:12:43.316961 (+3673197us) ts_tablet_manager.cc:1120] Deleting tablet 7461aa76a1af41cb9253285bda708f30
0503 02:12:43.316971 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.379597 (+ 62626us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x616000183454. stack: 00007f1c8dedd756 00007f1c8dedd510 00007f1c8cf1048d 0000000000955804 00000000009557cd 000000000095519a 00007f1c9974ba00 00007f1c9aa25b83 00007f1c9d4b957c 00007f1c9d4eb8d4 00007f1c9d4ccbc0 00007f1c9d4cc94d 0000000000a60bcc 00007f1c8df23b74 00007f1c8df2719b 00007f1c8df27018
0503 02:12:43.432422 (+ 52825us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.450975 (+ 18553us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.487350 (+ 36375us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142267136,"tablet-delete.queue_time_us":3672967}
I20240503 02:12:43.489156 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 752) took 3821 ms. Trace:
I20240503 02:12:43.489320 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 7461aa76a1af41cb9253285bda708f30 (table default.Table_4f9e55262006461d8fd7dd706e640469 [id=668afa5a4e424ea5b5398d690a79fa57]) successfully deleted
I20240503 02:12:43.490648 5115 rpcz_store.cc:276] 0503 02:12:39.667422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.667478 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:43.489045 (+3821567us) ts_tablet_manager.cc:1120] Deleting tablet 396b48865e954d05b9ba8084a2e7f29b
0503 02:12:43.489055 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.489139 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3821367}
I20240503 02:12:43.490988 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 753) took 3823 ms. Trace:
I20240503 02:12:43.491050 5115 rpcz_store.cc:276] 0503 02:12:39.667958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.667999 (+ 41us) service_pool.cc:225] Handling call
0503 02:12:43.490905 (+3822906us) ts_tablet_manager.cc:1120] Deleting tablet ea4dac17dca3442d8934ac357779b78a
0503 02:12:43.490914 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.490975 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3822749}
I20240503 02:12:43.491253 5115 tablet_replica.cc:330] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:43.491426 5115 raft_consensus.cc:2229] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.491693 6619 consensus_queue.cc:1035] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.490072 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:43.493609 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 396b48865e954d05b9ba8084a2e7f29b because the tablet was not found. No further retry: Not found: Tablet not found: 396b48865e954d05b9ba8084a2e7f29b
I20240503 02:12:43.493932 5106 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:43.493919 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ea4dac17dca3442d8934ac357779b78a because the tablet was not found. No further retry: Not found: Tablet not found: ea4dac17dca3442d8934ac357779b78a
I20240503 02:12:43.494051 5106 ts_tablet_manager.cc:1367] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
W20240503 02:12:43.494587 4746 leader_election.cc:341] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.494735 4746 leader_election.cc:302] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.494728 5106 raft_consensus.cc:348] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.494904 5106 raft_consensus.cc:374] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.494951 5106 raft_consensus.cc:729] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.495138 6595 raft_consensus.cc:2726] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.495204 5106 consensus_queue.cc:260] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.495707 5106 ts_tablet_manager.cc:1398] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.495980 5106 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.497412 5106 tablet_bootstrap.cc:654] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.497622 5114 tablet_replica.cc:330] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.497841 5114 raft_consensus.cc:2229] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.498025 5114 raft_consensus.cc:2258] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.498569 5114 ts_tablet_manager.cc:1875] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.494051 5115 raft_consensus.cc:2258] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.505257 5115 ts_tablet_manager.cc:1875] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:43.510879 4608 consensus_peers.cc:485] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3db94fb9d53044d796e2973143d888fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.512387 5159 ts_tablet_manager.cc:1888] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.515584 5115 ts_tablet_manager.cc:1888] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.515795 5115 log.cc:1196] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b91abaf51abb4c2ca65412c3771c1ef2
I20240503 02:12:43.516162 5115 ts_tablet_manager.cc:1909] T b91abaf51abb4c2ca65412c3771c1ef2 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.516535 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 754) took 3848 ms. Trace:
I20240503 02:12:43.516659 5115 rpcz_store.cc:276] 0503 02:12:39.668378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.668418 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:43.491214 (+3822796us) ts_tablet_manager.cc:1120] Deleting tablet b91abaf51abb4c2ca65412c3771c1ef2
0503 02:12:43.491221 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.510653 (+ 19432us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.515113 (+ 4460us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.516517 (+ 1404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3822632}
I20240503 02:12:43.517239 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b91abaf51abb4c2ca65412c3771c1ef2 (table default.Table_df633d54f27d4e71b052cbf0582d8ba2 [id=65c98763e63d4447b90a16f448bccffe]) successfully deleted
I20240503 02:12:43.517704 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 755) took 3848 ms. Trace:
I20240503 02:12:43.517822 5115 rpcz_store.cc:276] 0503 02:12:39.668805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.668843 (+ 38us) service_pool.cc:225] Handling call
0503 02:12:43.517607 (+3848764us) ts_tablet_manager.cc:1120] Deleting tablet e94031465ce7407dab0f90ad8809ac21
0503 02:12:43.517617 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.517689 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3848562}
W20240503 02:12:43.518311 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet e94031465ce7407dab0f90ad8809ac21 because the tablet was not found. No further retry: Not found: Tablet not found: e94031465ce7407dab0f90ad8809ac21
I20240503 02:12:43.518710 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 744) took 4004 ms. Trace:
I20240503 02:12:43.518823 5115 rpcz_store.cc:276] 0503 02:12:39.513788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.684553 (+170765us) service_pool.cc:225] Handling call
0503 02:12:43.518607 (+3834054us) ts_tablet_manager.cc:1120] Deleting tablet 5e534255f170416690c18ca6bd571117
0503 02:12:43.518616 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.518696 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3833428}
W20240503 02:12:43.519356 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5e534255f170416690c18ca6bd571117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e534255f170416690c18ca6bd571117 already in progress: creating tablet
I20240503 02:12:43.519948 5159 log.cc:1196] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/4b4972057fae45168756d7c28ddbe2bf
I20240503 02:12:43.520687 6695 raft_consensus.cc:482] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.520843 6695 raft_consensus.cc:504] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.521190 5114 ts_tablet_manager.cc:1888] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.521294 5114 log.cc:1196] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/7178760c21904207b6c8317093871f63
I20240503 02:12:43.521575 5114 ts_tablet_manager.cc:1909] T 7178760c21904207b6c8317093871f63 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.521642 6695 leader_election.cc:288] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.521708 5159 ts_tablet_manager.cc:1909] T 4b4972057fae45168756d7c28ddbe2bf P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.521843 5106 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.522028 5106 ts_tablet_manager.cc:1367] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:12:43.522290 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 813) took 2412 ms. Trace:
I20240503 02:12:43.522409 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.522416 5114 rpcz_store.cc:276] 0503 02:12:41.110024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.110073 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:43.497566 (+2387493us) ts_tablet_manager.cc:1120] Deleting tablet 7178760c21904207b6c8317093871f63
0503 02:12:43.497578 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.518167 (+ 20589us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.521089 (+ 2922us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.522270 (+ 1181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349854}
I20240503 02:12:43.522532 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 854) took 2396 ms. Trace:
I20240503 02:12:43.522600 5159 rpcz_store.cc:276] 0503 02:12:41.126297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.127226 (+ 929us) service_pool.cc:225] Handling call
0503 02:12:43.479085 (+2351859us) ts_tablet_manager.cc:1120] Deleting tablet 4b4972057fae45168756d7c28ddbe2bf
0503 02:12:43.479752 (+ 667us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.485181 (+ 5429us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.512246 (+ 27065us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.522511 (+ 10265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768,"tablet-delete.queue_time_us":2351567}
I20240503 02:12:43.522732 5106 raft_consensus.cc:348] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.522838 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 7178760c21904207b6c8317093871f63 (table default.Table_7d5419d4bbf34b67bcc05cf32663c2df [id=07e16e4056424605a70ccfaa25a24613]) successfully deleted
I20240503 02:12:43.523123 5114 tablet_replica.cc:330] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.523133 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.523339 4519 raft_consensus.cc:2454] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.522994 5106 raft_consensus.cc:374] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.523681 5106 raft_consensus.cc:729] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.523751 5103 tablet_bootstrap.cc:492] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.523716 4608 leader_election.cc:302] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.523999 6695 raft_consensus.cc:2781] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.523941 5106 consensus_queue.cc:260] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.524062 6695 raft_consensus.cc:482] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.524116 6695 raft_consensus.cc:3037] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.524400 5106 ts_tablet_manager.cc:1398] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.524689 5106 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.525627 5103 ts_tablet_manager.cc:1367] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.005s
I20240503 02:12:43.526075 5106 tablet_bootstrap.cc:654] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.526870 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 756) took 3826 ms. Trace:
I20240503 02:12:43.526818 6695 raft_consensus.cc:504] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.526939 5115 rpcz_store.cc:276] 0503 02:12:39.700110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.740969 (+ 40859us) service_pool.cc:225] Handling call
0503 02:12:43.526758 (+3785789us) ts_tablet_manager.cc:1120] Deleting tablet 9fa3df92b6e544de9ca76d98706eebec
0503 02:12:43.526765 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.526854 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3778448}
I20240503 02:12:43.527180 5115 tablet_replica.cc:330] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:43.527334 5115 raft_consensus.cc:2229] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.527482 5115 raft_consensus.cc:2258] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.527933 5115 ts_tablet_manager.cc:1875] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:43.528713 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9fa3df92b6e544de9ca76d98706eebec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fa3df92b6e544de9ca76d98706eebec already in progress: creating tablet
I20240503 02:12:43.523804 5114 raft_consensus.cc:2229] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.529197 5114 raft_consensus.cc:2258] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.527336 6695 leader_election.cc:288] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.529299 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:43.529464 4519 raft_consensus.cc:3037] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.529879 5114 ts_tablet_manager.cc:1875] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.522603 4811 raft_consensus.cc:2454] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.528409 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195767d80384a6c9a4690292f6776de" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.531239 4808 raft_consensus.cc:3037] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.531877 4519 raft_consensus.cc:2454] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.532514 4608 leader_election.cc:302] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.532766 6695 raft_consensus.cc:2781] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.532965 6695 raft_consensus.cc:686] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.533191 6695 consensus_queue.cc:237] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.533840 4808 raft_consensus.cc:2454] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.534654 5159 tablet_replica.cc:330] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.534611 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 4b4972057fae45168756d7c28ddbe2bf (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]) successfully deleted
I20240503 02:12:43.526257 5103 raft_consensus.cc:348] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.538267 5159 raft_consensus.cc:2229] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.538492 5103 raft_consensus.cc:374] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.539199 5103 raft_consensus.cc:729] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.539500 5159 raft_consensus.cc:2258] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.539950 5103 consensus_queue.cc:260] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.540774 5159 ts_tablet_manager.cc:1875] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.544121 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 3750506fa4034e2fa830a57c927023ee (DEFAULT_TABLE table=default.Table_1643ac26c14545c49e6ce2196a666402 [id=5509b3eeb7684292919ad14185494314]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.545369 5114 ts_tablet_manager.cc:1888] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.546644 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.546686 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6ce4d17d45734d5483fa95f40b455dd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.546861 5114 log.cc:1196] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/48e713ee37d042309166b0f118e14986
W20240503 02:12:43.547261 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ce4d17d45734d5483fa95f40b455dd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.547776 5114 ts_tablet_manager.cc:1909] T 48e713ee37d042309166b0f118e14986 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.548640 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 814) took 2438 ms. Trace:
I20240503 02:12:43.551105 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 6708c3a12ce94606ba5431bac1fdadf3 (DEFAULT_TABLE table=default.Table_c1106a95348a4a3f8961db51ddeb4221 [id=7afbcbb66e044e1caa45246694d89142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.551504 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6708c3a12ce94606ba5431bac1fdadf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.551633 5114 rpcz_store.cc:276] 0503 02:12:41.110414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.110463 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:43.523062 (+2412599us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:43.523070 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.532599 (+ 9529us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.545261 (+ 12662us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.548618 (+ 3357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375304}
I20240503 02:12:43.553718 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3750506fa4034e2fa830a57c927023ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.554165 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 48e713ee37d042309166b0f118e14986 (table default.Table_fb11bd168d9b4a22940b384400a5c6d5 [id=b4c6a45a47e647b1947d0d7f2ea30afa]) successfully deleted
I20240503 02:12:43.554549 5114 tablet_replica.cc:330] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.554790 5114 raft_consensus.cc:2229] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.555018 5114 raft_consensus.cc:2258] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.555581 5114 ts_tablet_manager.cc:1875] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.557199 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 3750506fa4034e2fa830a57c927023ee (DEFAULT_TABLE table=default.Table_1643ac26c14545c49e6ce2196a666402 [id=5509b3eeb7684292919ad14185494314]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.557596 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3750506fa4034e2fa830a57c927023ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.560880 6699 consensus_queue.cc:1035] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.561620 5105 tablet_bootstrap.cc:492] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.561740 5105 ts_tablet_manager.cc:1367] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.009s
I20240503 02:12:43.562337 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:43.562518 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:43.562453 5105 raft_consensus.cc:348] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.562697 5105 raft_consensus.cc:374] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.562767 5105 raft_consensus.cc:729] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.563025 5105 consensus_queue.cc:260] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.563832 5105 ts_tablet_manager.cc:1398] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:43.564154 5105 tablet_bootstrap.cc:492] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.544421 5103 ts_tablet_manager.cc:1398] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 02:12:43.549194 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 3750506fa4034e2fa830a57c927023ee (DEFAULT_TABLE table=default.Table_1643ac26c14545c49e6ce2196a666402 [id=5509b3eeb7684292919ad14185494314]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.566155 5105 tablet_bootstrap.cc:654] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.566267 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3750506fa4034e2fa830a57c927023ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.568435 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.568658 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 7461aa76a1af41cb9253285bda708f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.568832 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 43119471f91c4a1ba900f267c88db9a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 368aabce19984d03aa7116c8df990ef1) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.572727 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.573113 5103 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.573253 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 6708c3a12ce94606ba5431bac1fdadf3 (DEFAULT_TABLE table=default.Table_c1106a95348a4a3f8961db51ddeb4221 [id=7afbcbb66e044e1caa45246694d89142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.573640 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6708c3a12ce94606ba5431bac1fdadf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.573885 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b01907833d44f6a000a4de3a2a6e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.574256 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7afb429d64497abcd9f1d0910651c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.574399 5103 tablet_bootstrap.cc:654] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.581724 6620 consensus_queue.cc:1035] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.542949 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 6708c3a12ce94606ba5431bac1fdadf3 (DEFAULT_TABLE table=default.Table_c1106a95348a4a3f8961db51ddeb4221 [id=7afbcbb66e044e1caa45246694d89142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.589100 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6708c3a12ce94606ba5431bac1fdadf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.603593 5105 tablet_bootstrap.cc:492] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.606122 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 2f62b4441e694385a4403cc13fc15b2a (DEFAULT_TABLE table=default.Table_81c7b0a5ee98426c9e050f8f5f8d7d35 [id=38f010ec43ce4888914a1e477254db04]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:43.608692 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 442d795ee01c417dbdeda41fd6853dd7 consensus state has no leader
I20240503 02:12:43.611209 4272 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:43.611501 4272 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:43.613446 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cf51dfd849f741bcb578d641aee31de4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.608727 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 2f62b4441e694385a4403cc13fc15b2a (DEFAULT_TABLE table=default.Table_81c7b0a5ee98426c9e050f8f5f8d7d35 [id=38f010ec43ce4888914a1e477254db04]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:43.614025 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf51dfd849f741bcb578d641aee31de4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.614238 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f62b4441e694385a4403cc13fc15b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.615278 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 0794df2bfc5e45db95cf4c745892e695
I20240503 02:12:43.615896 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f88ecd4e6c544f43881964c3c4c128d9" } new_table_name: "default.Table_f10e78f12a3948bca598aef0a06d8e58" modify_external_catalogs: true
I20240503 02:12:43.611418 6595 raft_consensus.cc:482] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.617049 6595 raft_consensus.cc:504] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.617894 6595 leader_election.cc:288] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.618189 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 2f62b4441e694385a4403cc13fc15b2a (DEFAULT_TABLE table=default.Table_81c7b0a5ee98426c9e050f8f5f8d7d35 [id=38f010ec43ce4888914a1e477254db04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.618608 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f62b4441e694385a4403cc13fc15b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.618698 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66f88a24b74f2c9b2050af081d4601 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.619235 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 0794df2bfc5e45db95cf4c745892e695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.619285 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc2ebfb11aa4840adcc35da73be75a2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:43.619778 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f62b4441e694385a4403cc13fc15b2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:43.620108 4746 leader_election.cc:341] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.620630 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc2ebfb11aa4840adcc35da73be75a2" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.621148 4746 leader_election.cc:341] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.621259 4746 leader_election.cc:302] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.621655 6595 raft_consensus.cc:2726] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.621896 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 0794df2bfc5e45db95cf4c745892e695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.618202 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:43.627912 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a341379382784488afd99fb235b712f5" } new_table_name: "default.Table_b7945bd218d44cc79da4a42ae9d99679" modify_external_catalogs: true
I20240503 02:12:43.630245 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b173d78ba74ede801b12cccbeef6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.630618 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.630841 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet b38d63d2cc8c4f54ba43cdf8cde1841f with delete_type TABLET_DATA_DELETED (replaced by tablet 54959b3e30944fab8bd5fd47ce42d246) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.633034 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195767d80384a6c9a4690292f6776de with delete_type TABLET_DATA_DELETED (replaced by tablet b30e440f79a64e04b7e527b52299301f) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:43.636319 5105 ts_tablet_manager.cc:1367] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.004s
I20240503 02:12:43.637409 5105 raft_consensus.cc:348] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.637709 5105 raft_consensus.cc:374] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.637787 5105 raft_consensus.cc:729] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.646934 5103 tablet_bootstrap.cc:492] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.647118 5103 ts_tablet_manager.cc:1367] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20240503 02:12:43.647428 5159 ts_tablet_manager.cc:1888] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.647554 5159 log.cc:1196] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2f862891b43a4845b723bfdb9a7da591
I20240503 02:12:43.647871 5159 ts_tablet_manager.cc:1909] T 2f862891b43a4845b723bfdb9a7da591 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.647827 5103 raft_consensus.cc:348] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.648036 5103 raft_consensus.cc:374] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.648102 5103 raft_consensus.cc:729] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.648306 5103 consensus_queue.cc:260] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.648592 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 855) took 2520 ms. Trace:
I20240503 02:12:43.648696 5159 rpcz_store.cc:276] 0503 02:12:41.127910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.127958 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:43.534602 (+2406644us) ts_tablet_manager.cc:1120] Deleting tablet 2f862891b43a4845b723bfdb9a7da591
0503 02:12:43.534614 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.552363 (+ 17749us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.647337 (+ 94974us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.648556 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406433}
I20240503 02:12:43.648967 5103 ts_tablet_manager.cc:1398] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.649225 5105 consensus_queue.cc:260] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.649953 5114 ts_tablet_manager.cc:1888] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.650068 5114 log.cc:1196] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/59855ab3e42b4e019b769007ccbbc360
I20240503 02:12:43.650434 5114 ts_tablet_manager.cc:1909] T 59855ab3e42b4e019b769007ccbbc360 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.650658 5105 ts_tablet_manager.cc:1398] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:43.651183 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 810) took 2542 ms. Trace:
I20240503 02:12:43.651276 5114 rpcz_store.cc:276] 0503 02:12:41.108854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.108904 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:43.554499 (+2445595us) ts_tablet_manager.cc:1120] Deleting tablet 59855ab3e42b4e019b769007ccbbc360
0503 02:12:43.554511 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.558492 (+ 3981us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.649855 (+ 91363us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.651118 (+ 1263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406634}
I20240503 02:12:43.651592 5114 tablet_replica.cc:330] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.651758 5114 raft_consensus.cc:2229] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:43.652070 5114 raft_consensus.cc:2258] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.652225 5105 tablet_bootstrap.cc:492] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.652614 5114 ts_tablet_manager.cc:1875] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.656124 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_54079b9b61d744c9aca36afd92498b47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:43.656728 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54079b9b61d744c9aca36afd92498b47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.658737 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 8b423a704ce24c358a05857da3915671 (DEFAULT_TABLE table=default.Table_7b0e58e30571429b92a41817909e236f [id=5f2c2453e4f9441486e14bec627841ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.659250 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b423a704ce24c358a05857da3915671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.659436 5105 tablet_bootstrap.cc:654] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.660143 5103 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.661583 5103 tablet_bootstrap.cc:654] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.662325 4507 tablet_service.cc:1467] Processing CreateTablet for tablet aaca026a347c470fb5e1e8e8ed13814d (DEFAULT_TABLE table=default.Table_1abeaf665dc64f07820a1729b1a0e01d [id=e6d2f198d1594626979e95c7df739919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.662736 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaca026a347c470fb5e1e8e8ed13814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.663353 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 493fdc3e09cd48ac879e00efa09b8a86 (DEFAULT_TABLE table=default.Table_e28d38896e194e4eb84ca9cefc1b9835 [id=a1af058366114055b891e4f2a8b1966e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.663686 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493fdc3e09cd48ac879e00efa09b8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.665283 4775 tablet_service.cc:1467] Processing CreateTablet for tablet aaca026a347c470fb5e1e8e8ed13814d (DEFAULT_TABLE table=default.Table_1abeaf665dc64f07820a1729b1a0e01d [id=e6d2f198d1594626979e95c7df739919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.665833 5103 tablet_bootstrap.cc:492] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.666015 5103 ts_tablet_manager.cc:1367] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:43.666757 5103 raft_consensus.cc:348] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.667043 5103 raft_consensus.cc:374] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.667145 5103 raft_consensus.cc:729] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.667423 5103 consensus_queue.cc:260] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.670789 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 8b423a704ce24c358a05857da3915671 (DEFAULT_TABLE table=default.Table_7b0e58e30571429b92a41817909e236f [id=5f2c2453e4f9441486e14bec627841ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.671517 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaca026a347c470fb5e1e8e8ed13814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.672672 5114 ts_tablet_manager.cc:1888] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.673492 5114 log.cc:1196] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/d39f042afb954daa923502ef12377bd5
I20240503 02:12:43.673626 5103 ts_tablet_manager.cc:1398] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:12:43.673911 5114 ts_tablet_manager.cc:1909] T d39f042afb954daa923502ef12377bd5 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.675127 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:43.675397 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:43.675629 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:43.675793 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:43.676025 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet a73ec6543b9a468492f3b97772ce43f9 (table default.Table_2aec4967b23a414596b8b354cf8de2d9 [id=eb724fd2c11544ab9d8d539d30a2ffa4]). Expected version 2 got 0
I20240503 02:12:43.676330 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 811) took 2567 ms. Trace:
I20240503 02:12:43.676487 5114 rpcz_store.cc:276] 0503 02:12:41.109358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.111738 (+ 2380us) service_pool.cc:225] Handling call
0503 02:12:43.651548 (+2539810us) ts_tablet_manager.cc:1120] Deleting tablet d39f042afb954daa923502ef12377bd5
0503 02:12:43.651557 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.668030 (+ 16473us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.672588 (+ 4558us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.676310 (+ 3722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2471680}
I20240503 02:12:43.677295 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b423a704ce24c358a05857da3915671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.677605 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 815) took 2467 ms. Trace:
I20240503 02:12:43.677728 5114 rpcz_store.cc:276] 0503 02:12:41.210072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.210199 (+ 127us) service_pool.cc:225] Handling call
0503 02:12:43.677487 (+2467288us) ts_tablet_manager.cc:1120] Deleting tablet 0c3b9a888b894897b7cc3cf08402febd
0503 02:12:43.677495 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.677588 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467090}
I20240503 02:12:43.678857 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 706120be4d2a468780ffd64ea54849a3 (DEFAULT_TABLE table=default.Table_a341379382784488afd99fb235b712f5 [id=2a643963d3f24df09af9788abde4b5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.679210 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706120be4d2a468780ffd64ea54849a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.680090 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 493fdc3e09cd48ac879e00efa09b8a86 (DEFAULT_TABLE table=default.Table_e28d38896e194e4eb84ca9cefc1b9835 [id=a1af058366114055b891e4f2a8b1966e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.680359 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493fdc3e09cd48ac879e00efa09b8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.682282 5159 tablet_replica.cc:330] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.682504 5159 raft_consensus.cc:2229] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.678223 5103 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.682680 5159 raft_consensus.cc:2258] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.678534 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 706120be4d2a468780ffd64ea54849a3 (DEFAULT_TABLE table=default.Table_a341379382784488afd99fb235b712f5 [id=2a643963d3f24df09af9788abde4b5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.683235 5159 ts_tablet_manager.cc:1875] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.683291 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706120be4d2a468780ffd64ea54849a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.686219 5103 tablet_bootstrap.cc:654] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.689067 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8740e2e0d443d88a2cbd624341edea (DEFAULT_TABLE table=default.Table_15da6a828615427d9059c958249befbc [id=50f14722fc7147a989be055808832c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.689415 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8740e2e0d443d88a2cbd624341edea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.690596 5105 tablet_bootstrap.cc:492] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.690735 5105 ts_tablet_manager.cc:1367] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20240503 02:12:43.691011 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 816) took 2475 ms. Trace:
I20240503 02:12:43.691085 5114 rpcz_store.cc:276] 0503 02:12:41.215782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.216100 (+ 318us) service_pool.cc:225] Handling call
0503 02:12:43.690903 (+2474803us) ts_tablet_manager.cc:1120] Deleting tablet 28c1a81589df496b92b980353f00e594
0503 02:12:43.690910 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.690997 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474615}
I20240503 02:12:43.691313 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 8b423a704ce24c358a05857da3915671 (DEFAULT_TABLE table=default.Table_7b0e58e30571429b92a41817909e236f [id=5f2c2453e4f9441486e14bec627841ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.691520 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 817) took 2461 ms. Trace:
I20240503 02:12:43.691586 5114 rpcz_store.cc:276] 0503 02:12:41.229588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.229646 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:43.691428 (+2461782us) ts_tablet_manager.cc:1120] Deleting tablet 5f8656a886564f06931039b516162b51
0503 02:12:43.691436 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.691508 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461564}
I20240503 02:12:43.691606 5105 raft_consensus.cc:348] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.691772 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b423a704ce24c358a05857da3915671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.691820 5105 raft_consensus.cc:374] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.691888 5105 raft_consensus.cc:729] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.691931 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 821) took 2451 ms. Trace:
I20240503 02:12:43.691987 5114 rpcz_store.cc:276] 0503 02:12:41.240081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.240138 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:43.691847 (+2451709us) ts_tablet_manager.cc:1120] Deleting tablet 39c48b14bc624c28a871d45e9429ee30
0503 02:12:43.691856 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.691919 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2451486}
I20240503 02:12:43.692090 5105 consensus_queue.cc:260] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.692775 5105 ts_tablet_manager.cc:1398] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.696004 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 5229955d223c4e5e90ad9506819ff524 (DEFAULT_TABLE table=default.Table_87ee6bd84c764979a8df9a93bdb3d424 [id=7a59990058a742f2a803f0d5f7ef9337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.696375 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5229955d223c4e5e90ad9506819ff524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.697011 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 706120be4d2a468780ffd64ea54849a3 (DEFAULT_TABLE table=default.Table_a341379382784488afd99fb235b712f5 [id=2a643963d3f24df09af9788abde4b5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.697430 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706120be4d2a468780ffd64ea54849a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.697451 5105 tablet_bootstrap.cc:492] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:43.700417 5159 ts_tablet_manager.cc:1888] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.700548 5159 log.cc:1196] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/607e49cde2fd4aab9f44921d3c3921a5
I20240503 02:12:43.701712 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 493fdc3e09cd48ac879e00efa09b8a86 (DEFAULT_TABLE table=default.Table_e28d38896e194e4eb84ca9cefc1b9835 [id=a1af058366114055b891e4f2a8b1966e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.702103 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493fdc3e09cd48ac879e00efa09b8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.692389 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 819) took 2454 ms. Trace:
I20240503 02:12:43.702306 5114 rpcz_store.cc:276] 0503 02:12:41.237475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.244976 (+ 7501us) service_pool.cc:225] Handling call
0503 02:12:43.692297 (+2447321us) ts_tablet_manager.cc:1120] Deleting tablet 3c421694ce1e424da7953d061a6bca3e
0503 02:12:43.692306 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.692373 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2447023}
I20240503 02:12:43.702481 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 5229955d223c4e5e90ad9506819ff524 (DEFAULT_TABLE table=default.Table_87ee6bd84c764979a8df9a93bdb3d424 [id=7a59990058a742f2a803f0d5f7ef9337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.702863 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5229955d223c4e5e90ad9506819ff524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.703249 5105 tablet_bootstrap.cc:654] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.705096 4646 tablet_service.cc:1467] Processing CreateTablet for tablet aaca026a347c470fb5e1e8e8ed13814d (DEFAULT_TABLE table=default.Table_1abeaf665dc64f07820a1729b1a0e01d [id=e6d2f198d1594626979e95c7df739919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.705489 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaca026a347c470fb5e1e8e8ed13814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.706926 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 820) took 2467 ms. Trace:
I20240503 02:12:43.707013 5114 rpcz_store.cc:276] 0503 02:12:41.239326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.241771 (+ 2445us) service_pool.cc:225] Handling call
0503 02:12:43.706788 (+2465017us) ts_tablet_manager.cc:1120] Deleting tablet efd8e0353b5f4ce484ac81d646b5323d
0503 02:12:43.706799 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.706905 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2356851}
I20240503 02:12:43.707554 5114 tablet_replica.cc:330] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.707738 5114 raft_consensus.cc:2229] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.707904 5114 raft_consensus.cc:2258] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.709412 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 09acc937f55944e5adea8b2ce1a46922 (DEFAULT_TABLE table=default.Table_3b36c6850376429a8ac96f0a2595db44 [id=b4998f85b6c34243af58699cee91413a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.709663 5114 ts_tablet_manager.cc:1875] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.709823 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09acc937f55944e5adea8b2ce1a46922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.709717 4500 tablet_service.cc:1467] Processing CreateTablet for tablet d6caafb7adff4f91a6a5756769eebde0 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.710182 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6caafb7adff4f91a6a5756769eebde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.711349 4637 tablet_service.cc:1467] Processing CreateTablet for tablet d6caafb7adff4f91a6a5756769eebde0 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.711750 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6caafb7adff4f91a6a5756769eebde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.713809 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8740e2e0d443d88a2cbd624341edea (DEFAULT_TABLE table=default.Table_15da6a828615427d9059c958249befbc [id=50f14722fc7147a989be055808832c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.714320 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8740e2e0d443d88a2cbd624341edea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.717289 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 5229955d223c4e5e90ad9506819ff524 (DEFAULT_TABLE table=default.Table_87ee6bd84c764979a8df9a93bdb3d424 [id=7a59990058a742f2a803f0d5f7ef9337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.717751 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5229955d223c4e5e90ad9506819ff524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.717695 4788 tablet_service.cc:1467] Processing CreateTablet for tablet d6caafb7adff4f91a6a5756769eebde0 (DEFAULT_TABLE table=default.Table_2fb716cd099049f99df15bdb22fe90fb [id=5a456a26bfba42b9bafe8e7d69c1c278]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.718164 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6caafb7adff4f91a6a5756769eebde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.727224 5159 ts_tablet_manager.cc:1909] T 607e49cde2fd4aab9f44921d3c3921a5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.729804 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 856) took 2601 ms. Trace:
I20240503 02:12:43.730444 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 452d494e14c440e6805055ba333b4e7f (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.730870 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d494e14c440e6805055ba333b4e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.731307 5159 rpcz_store.cc:276] 0503 02:12:41.128395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.128436 (+ 41us) service_pool.cc:225] Handling call
0503 02:12:43.682236 (+2553800us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:43.682246 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.696629 (+ 14383us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.700314 (+ 3685us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.729760 (+ 29446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553612}
I20240503 02:12:43.732400 5159 tablet_replica.cc:330] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.732434 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 09acc937f55944e5adea8b2ce1a46922 (DEFAULT_TABLE table=default.Table_3b36c6850376429a8ac96f0a2595db44 [id=b4998f85b6c34243af58699cee91413a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.732607 5159 raft_consensus.cc:2229] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.732774 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09acc937f55944e5adea8b2ce1a46922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.733775 5159 raft_consensus.cc:2258] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.734357 5159 ts_tablet_manager.cc:1875] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.736281 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 09acc937f55944e5adea8b2ce1a46922 (DEFAULT_TABLE table=default.Table_3b36c6850376429a8ac96f0a2595db44 [id=b4998f85b6c34243af58699cee91413a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.736662 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09acc937f55944e5adea8b2ce1a46922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.737640 5114 ts_tablet_manager.cc:1888] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.738353 5114 log.cc:1196] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/8a90ebcf29004d1a925b69fa20998dd3
I20240503 02:12:43.738765 5114 ts_tablet_manager.cc:1909] T 8a90ebcf29004d1a925b69fa20998dd3 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.739545 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 844) took 2123 ms. Trace:
I20240503 02:12:43.739722 5114 rpcz_store.cc:276] 0503 02:12:41.615796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.616178 (+ 382us) service_pool.cc:225] Handling call
0503 02:12:43.707510 (+2091332us) ts_tablet_manager.cc:1120] Deleting tablet 8a90ebcf29004d1a925b69fa20998dd3
0503 02:12:43.707518 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.717932 (+ 10414us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.737557 (+ 19625us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.739523 (+ 1966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2091144}
I20240503 02:12:43.740404 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8740e2e0d443d88a2cbd624341edea (DEFAULT_TABLE table=default.Table_15da6a828615427d9059c958249befbc [id=50f14722fc7147a989be055808832c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.740509 5114 tablet_replica.cc:330] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.740773 5114 raft_consensus.cc:2229] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.740828 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8740e2e0d443d88a2cbd624341edea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.741037 5114 raft_consensus.cc:2258] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.741642 5114 ts_tablet_manager.cc:1875] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.742302 5106 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.747843 5114 ts_tablet_manager.cc:1888] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.748057 5114 log.cc:1196] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/ea5e4db5204e4acf8719100d02d533fb
I20240503 02:12:43.748528 5114 ts_tablet_manager.cc:1909] T ea5e4db5204e4acf8719100d02d533fb P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.749102 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 0794df2bfc5e45db95cf4c745892e695 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.749470 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 845) took 1912 ms. Trace:
I20240503 02:12:43.749639 5114 rpcz_store.cc:276] 0503 02:12:41.837280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.838195 (+ 915us) service_pool.cc:225] Handling call
0503 02:12:43.740464 (+1902269us) ts_tablet_manager.cc:1120] Deleting tablet ea5e4db5204e4acf8719100d02d533fb
0503 02:12:43.740473 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.744836 (+ 4363us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.747748 (+ 2912us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.749445 (+ 1697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859422}
I20240503 02:12:43.753916 5114 tablet_replica.cc:330] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
W20240503 02:12:43.754056 4608 consensus_peers.cc:485] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59855ab3e42b4e019b769007ccbbc360. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.754204 5114 raft_consensus.cc:2229] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.754410 5114 raft_consensus.cc:2258] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.755731 4781 tablet_service.cc:1467] Processing CreateTablet for tablet cc0ea917e51e48ae8c8395af7ed85cfe (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.756309 4497 tablet_service.cc:1467] Processing CreateTablet for tablet df3b4151bc0440fca23ead599370f479 (DEFAULT_TABLE table=default.Table_8c58e481444e4e98b065e88e715ba236 [id=a2ce23904305497996fb6e474037d824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.756412 4778 tablet_service.cc:1467] Processing CreateTablet for tablet df3b4151bc0440fca23ead599370f479 (DEFAULT_TABLE table=default.Table_8c58e481444e4e98b065e88e715ba236 [id=a2ce23904305497996fb6e474037d824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.756615 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 59855ab3e42b4e019b769007ccbbc360 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
I20240503 02:12:43.756718 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3b4151bc0440fca23ead599370f479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.757076 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet d39f042afb954daa923502ef12377bd5 (table default.Table_2c388fbe8fd148dc93f0d292381c0363 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]) successfully deleted
W20240503 02:12:43.757485 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0c3b9a888b894897b7cc3cf08402febd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c3b9a888b894897b7cc3cf08402febd already in progress: creating tablet
W20240503 02:12:43.757826 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 28c1a81589df496b92b980353f00e594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28c1a81589df496b92b980353f00e594 already in progress: creating tablet
W20240503 02:12:43.758033 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5f8656a886564f06931039b516162b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8656a886564f06931039b516162b51 already in progress: creating tablet
W20240503 02:12:43.758258 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 39c48b14bc624c28a871d45e9429ee30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39c48b14bc624c28a871d45e9429ee30 already in progress: creating tablet
W20240503 02:12:43.758459 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3c421694ce1e424da7953d061a6bca3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c421694ce1e424da7953d061a6bca3e already in progress: creating tablet
W20240503 02:12:43.758666 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet efd8e0353b5f4ce484ac81d646b5323d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd8e0353b5f4ce484ac81d646b5323d already in progress: creating tablet
I20240503 02:12:43.758803 6698 raft_consensus.cc:482] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.759245 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 8a90ebcf29004d1a925b69fa20998dd3 (table default.Table_114d4d6efe8947578cff18d0c2f235d2 [id=41995c76abc744c1b75efd4e1a4b4383]) successfully deleted
I20240503 02:12:43.759454 4494 tablet_service.cc:1467] Processing CreateTablet for tablet d0ad7fbea59041a0b079ef7557d1ab34 (DEFAULT_TABLE table=default.Table_555e1f783a5f411999e6a287d62c7d18 [id=f793a47f154d4885af2d414a6a737e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.759728 4788 tablet_service.cc:1467] Processing CreateTablet for tablet d0ad7fbea59041a0b079ef7557d1ab34 (DEFAULT_TABLE table=default.Table_555e1f783a5f411999e6a287d62c7d18 [id=f793a47f154d4885af2d414a6a737e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.759891 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ad7fbea59041a0b079ef7557d1ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.759855 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet ea5e4db5204e4acf8719100d02d533fb (table default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]) successfully deleted
I20240503 02:12:43.760166 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 452d494e14c440e6805055ba333b4e7f (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.760308 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 2f862891b43a4845b723bfdb9a7da591 (table default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]) successfully deleted
I20240503 02:12:43.760910 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 607e49cde2fd4aab9f44921d3c3921a5 (table default.Table_0e5ff92986ff447dbc2c9e69a628e8fe [id=08bfe029472248d7b17aaa9a8ee63472]) successfully deleted
I20240503 02:12:43.760957 4500 tablet_service.cc:1467] Processing CreateTablet for tablet cc0ea917e51e48ae8c8395af7ed85cfe (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.761662 5114 ts_tablet_manager.cc:1875] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.761945 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0ea917e51e48ae8c8395af7ed85cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.761945 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 452d494e14c440e6805055ba333b4e7f (DEFAULT_TABLE table=default.Table_5d2a8af7b53948258ce2c6f305f318cb [id=3f70407631dd4d3c9a303411aaeaeaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.762367 6698 raft_consensus.cc:504] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.763891 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3b4151bc0440fca23ead599370f479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.763959 4651 tablet_service.cc:1467] Processing CreateTablet for tablet df3b4151bc0440fca23ead599370f479 (DEFAULT_TABLE table=default.Table_8c58e481444e4e98b065e88e715ba236 [id=a2ce23904305497996fb6e474037d824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.764564 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.764773 4518 raft_consensus.cc:2454] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:43.765412 4462 consensus_peers.cc:485] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43119471f91c4a1ba900f267c88db9a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.765471 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ad7fbea59041a0b079ef7557d1ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.766105 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d494e14c440e6805055ba333b4e7f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:43.766201 4608 consensus_peers.cc:485] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a90ebcf29004d1a925b69fa20998dd3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.766427 4608 leader_election.cc:302] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.766700 6695 raft_consensus.cc:2781] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.766772 6695 raft_consensus.cc:482] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.766830 6695 raft_consensus.cc:3037] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.769690 5106 ts_tablet_manager.cc:1367] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.218s user 0.005s sys 0.000s
I20240503 02:12:43.770491 5106 raft_consensus.cc:348] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.770733 5106 raft_consensus.cc:374] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.770820 5106 raft_consensus.cc:729] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.771049 5106 consensus_queue.cc:260] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.771718 5106 ts_tablet_manager.cc:1398] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.771978 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0ea917e51e48ae8c8395af7ed85cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.772099 5106 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.772547 4646 tablet_service.cc:1467] Processing CreateTablet for tablet cc0ea917e51e48ae8c8395af7ed85cfe (DEFAULT_TABLE table=default.Table_71333c41521440e29744c283c842b895 [id=2c53f7ca73224652a86aa693b69b064c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.773299 6698 leader_election.cc:288] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.773551 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d494e14c440e6805055ba333b4e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.773864 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3b4151bc0440fca23ead599370f479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.775578 6695 raft_consensus.cc:504] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.776181 6695 leader_election.cc:288] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.783030 6595 raft_consensus.cc:482] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.783213 6595 raft_consensus.cc:504] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.783902 6595 leader_election.cc:288] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.784837 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:43.785076 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:43.785270 4519 raft_consensus.cc:3037] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.785406 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.785790 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0ea917e51e48ae8c8395af7ed85cfe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:43.785965 4746 leader_election.cc:341] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:43.786190 4746 leader_election.cc:341] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.786296 4746 leader_election.cc:302] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.786691 6595 raft_consensus.cc:2726] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.787910 4519 raft_consensus.cc:2454] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.788360 4608 leader_election.cc:302] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:43.788599 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:43.788635 6695 raft_consensus.cc:2781] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.788839 6695 raft_consensus.cc:686] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.788847 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:43.789089 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc]). Expected version 2 got 0
I20240503 02:12:43.789068 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.789116 6695 consensus_queue.cc:237] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.789263 4806 raft_consensus.cc:3037] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.791932 4806 raft_consensus.cc:2454] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.792634 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b8db679ba24ac6a93be591c0513ab0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.792840 4808 raft_consensus.cc:2362] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1. Re-sending same reply.
I20240503 02:12:43.801342 4647 tablet_service.cc:1467] Processing CreateTablet for tablet d0ad7fbea59041a0b079ef7557d1ab34 (DEFAULT_TABLE table=default.Table_555e1f783a5f411999e6a287d62c7d18 [id=f793a47f154d4885af2d414a6a737e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.801445 5159 ts_tablet_manager.cc:1888] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.801714 4507 tablet_service.cc:1467] Processing CreateTablet for tablet ff1275c9ea5c4ef0b9ebcdf919abf791 (DEFAULT_TABLE table=default.Table_bdfc07f1b4c94c97b1eddb89795526c7 [id=c85e521dd2834253ba9634c5d8f0bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.801805 5159 log.cc:1196] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:43.802094 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1275c9ea5c4ef0b9ebcdf919abf791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.802511 5159 ts_tablet_manager.cc:1909] T c1f39e8d17b44895bd5c979d82145ad9 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.803568 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 857) took 2674 ms. Trace:
I20240503 02:12:43.804755 5159 rpcz_store.cc:276] 0503 02:12:41.128850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.128893 (+ 43us) service_pool.cc:225] Handling call
0503 02:12:43.732324 (+2603431us) ts_tablet_manager.cc:1120] Deleting tablet c1f39e8d17b44895bd5c979d82145ad9
0503 02:12:43.732334 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.751727 (+ 19393us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.801319 (+ 49592us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.803529 (+ 2210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2603204}
I20240503 02:12:43.805680 5159 tablet_replica.cc:330] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.806123 5159 raft_consensus.cc:2229] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.806588 5159 raft_consensus.cc:2258] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.807368 5106 tablet_bootstrap.cc:654] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.808135 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ccb80b3d1e0a41e282ba3a1039347590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.807971 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ad7fbea59041a0b079ef7557d1ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.808552 5114 ts_tablet_manager.cc:1888] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.808669 5114 log.cc:1196] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b9511e4de6c942b9b9e0790c88d3d543
W20240503 02:12:43.808770 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccb80b3d1e0a41e282ba3a1039347590 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:43.811213 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 50085.7us
I20240503 02:12:43.811654 5103 tablet_bootstrap.cc:492] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.811812 5103 ts_tablet_manager.cc:1367] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.003s
I20240503 02:12:43.812465 5103 raft_consensus.cc:348] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.812682 5103 raft_consensus.cc:374] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.812757 5103 raft_consensus.cc:729] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.813009 5103 consensus_queue.cc:260] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.814749 5115 ts_tablet_manager.cc:1888] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.815385 5103 ts_tablet_manager.cc:1398] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:43.815701 5103 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.817199 5103 tablet_bootstrap.cc:654] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.822115 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.821867 5115 log.cc:1196] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:43.824442 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.825166 5114 ts_tablet_manager.cc:1909] T b9511e4de6c942b9b9e0790c88d3d543 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.825786 4644 tablet_service.cc:1467] Processing CreateTablet for tablet ff1275c9ea5c4ef0b9ebcdf919abf791 (DEFAULT_TABLE table=default.Table_bdfc07f1b4c94c97b1eddb89795526c7 [id=c85e521dd2834253ba9634c5d8f0bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.825992 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 846) took 1988 ms. Trace:
I20240503 02:12:43.826081 5114 rpcz_store.cc:276] 0503 02:12:41.837429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.838362 (+ 933us) service_pool.cc:225] Handling call
0503 02:12:43.753860 (+1915498us) ts_tablet_manager.cc:1120] Deleting tablet b9511e4de6c942b9b9e0790c88d3d543
0503 02:12:43.753870 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.801653 (+ 47783us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.808454 (+ 6801us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.825973 (+ 17519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872744}
I20240503 02:12:43.826299 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1275c9ea5c4ef0b9ebcdf919abf791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.829150 4776 tablet_service.cc:1467] Processing CreateTablet for tablet ff1275c9ea5c4ef0b9ebcdf919abf791 (DEFAULT_TABLE table=default.Table_bdfc07f1b4c94c97b1eddb89795526c7 [id=c85e521dd2834253ba9634c5d8f0bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.829193 4500 tablet_service.cc:1467] Processing CreateTablet for tablet b79e20441a2b402f856892d636658149 (DEFAULT_TABLE table=default.Table_c2cccebe93474d93903c9085efb5a837 [id=91edd004aa604771afedaa3d224cdad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.829563 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79e20441a2b402f856892d636658149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.829557 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1275c9ea5c4ef0b9ebcdf919abf791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.830636 4651 tablet_service.cc:1467] Processing CreateTablet for tablet b79e20441a2b402f856892d636658149 (DEFAULT_TABLE table=default.Table_c2cccebe93474d93903c9085efb5a837 [id=91edd004aa604771afedaa3d224cdad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.830992 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79e20441a2b402f856892d636658149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.831961 6595 raft_consensus.cc:482] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.832082 6595 raft_consensus.cc:504] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.832669 6595 leader_election.cc:288] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:43.833662 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.834333 4746 leader_election.cc:341] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.835688 4781 tablet_service.cc:1467] Processing CreateTablet for tablet b79e20441a2b402f856892d636658149 (DEFAULT_TABLE table=default.Table_c2cccebe93474d93903c9085efb5a837 [id=91edd004aa604771afedaa3d224cdad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.836990 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79e20441a2b402f856892d636658149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.831247 5115 ts_tablet_manager.cc:1909] T ea7afb429d64497abcd9f1d0910651c3 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:43.833442 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:43.839277 4746 leader_election.cc:341] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.839397 4746 leader_election.cc:302] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.839987 6595 raft_consensus.cc:2726] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.842245 5159 ts_tablet_manager.cc:1875] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.842576 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 151f925dcb304594b60c85d850b83533
I20240503 02:12:43.843067 4271 catalog_manager.cc:6554] ReplaceTablet: tablet 151f925dcb304594b60c85d850b83533 deleted and replaced by tablet 7833167a80c34f3e85bf9e79185ad84a
I20240503 02:12:43.843192 4271 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36560 (request call id 1121) took 1158 ms. Trace:
I20240503 02:12:43.843293 4271 rpcz_store.cc:276] 0503 02:12:42.684980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.196485 (+511505us) service_pool.cc:225] Handling call
0503 02:12:43.843158 (+646673us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:43.595834 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:43.595960 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:43.595967 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:43.595970 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:43.596100 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:43.596113 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:43.596118 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:43.596206 (+ 88us) tablet.cc:699] Row locks acquired
0503 02:12:43.596209 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:43.596249 (+ 40us) write_op.cc:270] Start()
0503 02:12:43.596287 (+ 38us) write_op.cc:276] Timestamp: P: 1714702363596184 usec, L: 0
0503 02:12:43.596291 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:43.596513 (+ 222us) log.cc:842] Serialized 622 byte log entry
0503 02:12:43.815187 (+218674us) op_driver.cc:444] REPLICATION: finished
0503 02:12:43.815604 (+ 417us) write_op.cc:301] APPLY: starting
0503 02:12:43.815624 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:43.815672 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:43.815705 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:43.815822 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:43.815885 (+ 63us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 02:12:43.815893 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:12:43.815977 (+ 84us) log.cc:842] Serialized 29 byte log entry
0503 02:12:43.816046 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:43.816068 (+ 22us) write_op.cc:454] Released schema lock
0503 02:12:43.816076 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:43.816129 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:43.816133 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":361,"apply.run_cpu_time_us":583,"apply.run_wall_time_us":582,"num_ops":2,"prepare.queue_time_us":51909,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":775,"replication_time_us":218871}]]}
I20240503 02:12:43.845639 5114 tablet_replica.cc:330] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.845881 5114 raft_consensus.cc:2229] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.845873 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b12b7f4b0c7048f38687ebc605d55be3" } new_table_name: "default.Table_aa9b15fc98e24fe9af93ac8c7d3a4d8a" modify_external_catalogs: true
I20240503 02:12:43.846065 5114 raft_consensus.cc:2258] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.846676 5114 ts_tablet_manager.cc:1875] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.847832 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet e82190dfb1b1418b9f20bf4d0c2c2409
I20240503 02:12:43.848433 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:43.850214 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 3ee32345fb9349f19af09a3447c4caad
I20240503 02:12:43.850904 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_aabd4d0faa1c4125805cfae21e766300" } new_table_name: "default.Table_040f5a34b777484280e6ffe1b0ba112f" modify_external_catalogs: true
I20240503 02:12:43.852507 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 2848c837598c49628c11374d3c875b3e
I20240503 02:12:43.853045 4266 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36560 (request call id 1123) took 1025 ms. Trace:
I20240503 02:12:43.853142 4266 rpcz_store.cc:276] 0503 02:12:42.827106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.198485 (+371379us) service_pool.cc:225] Handling call
0503 02:12:43.198946 (+ 461us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 02:12:43.199122 (+ 176us) catalog_manager.cc:2749] Modifying in-memory table state
0503 02:12:43.199191 (+ 69us) catalog_manager.cc:2758] Locking tablets
0503 02:12:43.544094 (+344903us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 02:12:43.818308 (+274214us) catalog_manager.cc:2790] Removing table from by-name map
0503 02:12:43.852436 (+ 34128us) catalog_manager.cc:2806] Committing in-memory state
0503 02:12:43.852482 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 02:12:43.852983 (+ 501us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:43.596638 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:43.596751 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:43.596757 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:12:43.596759 (+ 2us) tablet.cc:660] Decoding operations
0503 02:12:43.596854 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:43.596863 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:43.596868 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:12:43.596993 (+ 125us) tablet.cc:699] Row locks acquired
0503 02:12:43.596996 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:12:43.597026 (+ 30us) write_op.cc:270] Start()
0503 02:12:43.597054 (+ 28us) write_op.cc:276] Timestamp: P: 1714702363596964 usec, L: 0
0503 02:12:43.597058 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:12:43.597205 (+ 147us) log.cc:842] Serialized 725 byte log entry
0503 02:12:43.815255 (+218050us) op_driver.cc:444] REPLICATION: finished
0503 02:12:43.816214 (+ 959us) write_op.cc:301] APPLY: starting
0503 02:12:43.816225 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:43.816258 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:43.816282 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:43.816336 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:43.816362 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 02:12:43.816369 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:43.816428 (+ 59us) log.cc:842] Serialized 29 byte log entry
0503 02:12:43.816470 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:43.816484 (+ 14us) write_op.cc:454] Released schema lock
0503 02:12:43.816490 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 02:12:43.816532 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:43.816536 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":931,"apply.run_cpu_time_us":360,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":52023,"prepare.run_cpu_time_us":631,"prepare.run_wall_time_us":631,"replication_time_us":218183}]]}
I20240503 02:12:43.854055 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0dc297ee46134fc6ba917b6c48a7a612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:43.854373 5103 tablet_bootstrap.cc:492] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.854537 5103 ts_tablet_manager.cc:1367] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
W20240503 02:12:43.854671 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dc297ee46134fc6ba917b6c48a7a612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:43.855391 5103 raft_consensus.cc:348] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.855598 5103 raft_consensus.cc:374] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.855646 5103 raft_consensus.cc:729] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.855854 5103 consensus_queue.cc:260] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.856426 5103 ts_tablet_manager.cc:1398] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:43.858503 5103 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:43.859979 5103 tablet_bootstrap.cc:654] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.865190 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:43.866135 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
W20240503 02:12:43.869778 4608 consensus_peers.cc:485] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.870211 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:43.870582 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:43.870887 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:43.873457 5106 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.873792 5106 ts_tablet_manager.cc:1367] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20240503 02:12:43.874825 4519 raft_consensus.cc:3037] T de4e96b4be094101bf74904b2d66b6c7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.874857 5106 raft_consensus.cc:348] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.875118 5106 raft_consensus.cc:374] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.875178 5106 raft_consensus.cc:729] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.875474 5159 ts_tablet_manager.cc:1888] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:43.875623 5159 log.cc:1196] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/79b8db679ba24ac6a93be591c0513ab0
I20240503 02:12:43.875443 5106 consensus_queue.cc:260] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.875947 5159 ts_tablet_manager.cc:1909] T 79b8db679ba24ac6a93be591c0513ab0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:43.876773 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 859) took 2736 ms. Trace:
I20240503 02:12:43.876866 5159 rpcz_store.cc:276] 0503 02:12:41.140104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.140160 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:43.805610 (+2665450us) ts_tablet_manager.cc:1120] Deleting tablet 79b8db679ba24ac6a93be591c0513ab0
0503 02:12:43.805619 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.851007 (+ 45388us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.875315 (+ 24308us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.876747 (+ 1432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2664532}
I20240503 02:12:43.881547 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 7381c072411940aaab724e9e5b28da3c (DEFAULT_TABLE table=default.Table_cec6bc0bc6bc45b495af097e7fdb43bd [id=59124f6164f44460b7b2d7735f91d02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.882124 4675 raft_consensus.cc:3037] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.883073 5159 tablet_replica.cc:330] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:43.883299 5159 raft_consensus.cc:2229] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.883479 5159 raft_consensus.cc:2258] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.884073 5159 ts_tablet_manager.cc:1875] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.885972 6560 consensus_queue.cc:1035] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.887117 6595 raft_consensus.cc:482] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.887267 6595 raft_consensus.cc:504] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.887599 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 7381c072411940aaab724e9e5b28da3c (DEFAULT_TABLE table=default.Table_cec6bc0bc6bc45b495af097e7fdb43bd [id=59124f6164f44460b7b2d7735f91d02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.887923 6595 leader_election.cc:288] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.888001 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381c072411940aaab724e9e5b28da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.888641 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf10c48d165047668fbd5ca17f917fe3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.889580 4746 leader_election.cc:341] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.890733 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf10c48d165047668fbd5ca17f917fe3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:43.891510 4746 leader_election.cc:341] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.891630 4746 leader_election.cc:302] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.892237 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 7381c072411940aaab724e9e5b28da3c (DEFAULT_TABLE table=default.Table_cec6bc0bc6bc45b495af097e7fdb43bd [id=59124f6164f44460b7b2d7735f91d02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.892690 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381c072411940aaab724e9e5b28da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.895512 4644 tablet_service.cc:1467] Processing CreateTablet for tablet c5ced39756f54ef8990ef3c2915eb1b1 (DEFAULT_TABLE table=default.Table_2e62fd1e33c643c98023e90fbb294162 [id=2c4e844d5d0a470c8278fbd76e88c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.895762 4494 tablet_service.cc:1467] Processing CreateTablet for tablet c5ced39756f54ef8990ef3c2915eb1b1 (DEFAULT_TABLE table=default.Table_2e62fd1e33c643c98023e90fbb294162 [id=2c4e844d5d0a470c8278fbd76e88c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.897182 4788 tablet_service.cc:1467] Processing CreateTablet for tablet c5ced39756f54ef8990ef3c2915eb1b1 (DEFAULT_TABLE table=default.Table_2e62fd1e33c643c98023e90fbb294162 [id=2c4e844d5d0a470c8278fbd76e88c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.897444 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1fac803dd84eee8722f0a14559faf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.897593 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ced39756f54ef8990ef3c2915eb1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.896174 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ced39756f54ef8990ef3c2915eb1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.898372 4497 tablet_service.cc:1467] Processing CreateTablet for tablet d01bba76cfd04c4e84042a2311f461a5 (DEFAULT_TABLE table=default.Table_c535e618b79044af988f914e012015f3 [id=5d04edf960624bbeac252fc8f24caa00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.901808 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01bba76cfd04c4e84042a2311f461a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.905247 5106 ts_tablet_manager.cc:1398] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 02:12:43.905489 6595 raft_consensus.cc:2726] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.905649 6595 raft_consensus.cc:482] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.905741 6595 raft_consensus.cc:504] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.906342 6595 leader_election.cc:288] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:43.906965 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 757) took 4041 ms. Trace:
I20240503 02:12:43.908686 5106 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.910588 6698 raft_consensus.cc:482] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.911063 6698 raft_consensus.cc:504] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.911329 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7381c072411940aaab724e9e5b28da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.911713 6698 leader_election.cc:288] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.911916 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ced39756f54ef8990ef3c2915eb1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.912220 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.912428 4808 raft_consensus.cc:2454] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.912833 4608 leader_election.cc:302] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:43.913144 6698 raft_consensus.cc:2781] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.913213 6698 raft_consensus.cc:482] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.913268 6698 raft_consensus.cc:3037] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.915896 6698 raft_consensus.cc:504] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.916600 6698 leader_election.cc:288] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.917119 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.917249 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:43.917372 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.917578 4808 raft_consensus.cc:3037] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:43.917732 4608 leader_election.cc:341] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:43.917945 4608 leader_election.cc:341] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.918139 6698 raft_consensus.cc:482] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.918270 6698 raft_consensus.cc:504] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.918836 6698 leader_election.cc:288] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.919584 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d6d3ca80c54c629a5de82b06a7aac4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:43.920076 4608 leader_election.cc:341] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.920212 4808 raft_consensus.cc:2454] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.920646 5106 tablet_bootstrap.cc:654] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.921223 4608 leader_election.cc:302] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:43.921661 4608 leader_election.cc:341] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:43.921751 4608 leader_election.cc:302] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.921976 6699 raft_consensus.cc:2781] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.922108 6699 raft_consensus.cc:686] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.922403 6698 raft_consensus.cc:2726] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.922338 6699 consensus_queue.cc:237] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.922999 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet c757a3a979c2489daa621cf8ea293ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:43.923537 4644 tablet_service.cc:1467] Processing CreateTablet for tablet d01bba76cfd04c4e84042a2311f461a5 (DEFAULT_TABLE table=default.Table_c535e618b79044af988f914e012015f3 [id=5d04edf960624bbeac252fc8f24caa00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.923942 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01bba76cfd04c4e84042a2311f461a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.925302 4788 tablet_service.cc:1467] Processing CreateTablet for tablet d01bba76cfd04c4e84042a2311f461a5 (DEFAULT_TABLE table=default.Table_c535e618b79044af988f914e012015f3 [id=5d04edf960624bbeac252fc8f24caa00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.926378 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01bba76cfd04c4e84042a2311f461a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.927188 6698 raft_consensus.cc:482] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.927462 6698 raft_consensus.cc:504] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.928197 6698 leader_election.cc:288] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.929044 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:43.929268 4813 raft_consensus.cc:2454] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:43.929884 4608 leader_election.cc:302] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:43.909559 5115 rpcz_store.cc:276] 0503 02:12:39.865217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.865299 (+ 82us) service_pool.cc:225] Handling call
0503 02:12:43.527144 (+3661845us) ts_tablet_manager.cc:1120] Deleting tablet ea7afb429d64497abcd9f1d0910651c3
0503 02:12:43.527151 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.535432 (+ 8281us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.814661 (+279229us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.906930 (+ 92269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661604}
I20240503 02:12:43.930181 5114 ts_tablet_manager.cc:1888] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:43.932214 5105 tablet_bootstrap.cc:492] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:43.933092 5105 ts_tablet_manager.cc:1367] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.236s user 0.005s sys 0.031s
I20240503 02:12:43.935220 5105 raft_consensus.cc:348] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.935868 5114 log.cc:1196] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/87713e899d2148308316d807a6d76921
I20240503 02:12:43.936432 5105 raft_consensus.cc:374] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.936753 5115 tablet_replica.cc:330] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:43.936842 5114 ts_tablet_manager.cc:1909] T 87713e899d2148308316d807a6d76921 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:43.936993 5115 raft_consensus.cc:2229] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.937191 5115 raft_consensus.cc:2258] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.937800 5115 ts_tablet_manager.cc:1875] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.937184 5105 raft_consensus.cc:729] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.938306 5105 consensus_queue.cc:260] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.930241 6698 raft_consensus.cc:2781] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.940068 5105 ts_tablet_manager.cc:1398] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240503 02:12:43.939591 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 48061.5us
I20240503 02:12:43.940232 6698 raft_consensus.cc:482] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.940346 6698 raft_consensus.cc:3037] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.941069 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:43.942811 6698 raft_consensus.cc:504] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:43.942950 4746 leader_election.cc:341] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.943845 6698 leader_election.cc:288] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:43.944139 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:43.944347 4813 raft_consensus.cc:3037] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.947096 4813 raft_consensus.cc:2454] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.947840 4608 leader_election.cc:302] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:43.951050 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.951534 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:43.951711 4527 raft_consensus.cc:3037] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:43.953084 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cf10c48d165047668fbd5ca17f917fe3 consensus state has no leader
I20240503 02:12:43.954825 4527 raft_consensus.cc:2454] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:43.955909 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f630bf6a5740758fa1f81477ec59d3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:43.956125 4518 raft_consensus.cc:2362] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1. Re-sending same reply.
I20240503 02:12:43.956231 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet ea7afb429d64497abcd9f1d0910651c3 (table default.Table_29408dd96a06477d99f95b091aa7f728 [id=770372fb7753483dba3373ef6f6966e8]) successfully deleted
I20240503 02:12:43.958098 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 847) took 2120 ms. Trace:
I20240503 02:12:43.958321 5114 rpcz_store.cc:276] 0503 02:12:41.837511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.838454 (+ 943us) service_pool.cc:225] Handling call
0503 02:12:43.845568 (+2007114us) ts_tablet_manager.cc:1120] Deleting tablet 87713e899d2148308316d807a6d76921
0503 02:12:43.845582 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.867478 (+ 21896us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.930014 (+ 62536us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.950905 (+ 20891us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x61800000fd18. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139331319d 00007f139331420e 00007f13a681a4ed 00007f13a67966b3 00007f13a28f99dd 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74
0503 02:12:43.958081 (+ 7176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28751616,"tablet-delete.queue_time_us":1964400}
I20240503 02:12:43.959034 6695 raft_consensus.cc:2781] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.959112 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 848) took 2120 ms. Trace:
I20240503 02:12:43.959184 6695 raft_consensus.cc:686] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:43.959194 5114 rpcz_store.cc:276] 0503 02:12:41.838738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.838785 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:43.958983 (+2120198us) ts_tablet_manager.cc:1120] Deleting tablet 70e08c10c15f41878202cef0c8d5ad3c
0503 02:12:43.958993 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.959097 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2077764}
I20240503 02:12:43.959595 5114 tablet_replica.cc:330] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:43.959451 6695 consensus_queue.cc:237] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.959772 5114 raft_consensus.cc:2229] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.959947 5114 raft_consensus.cc:2258] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:43.959864 4746 leader_election.cc:341] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:43.960110 4746 leader_election.cc:302] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:43.961529 6595 raft_consensus.cc:2726] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:43.964592 4858 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:43.966265 5114 ts_tablet_manager.cc:1875] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:43.969220 5105 tablet_bootstrap.cc:492] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:43.969810 4608 consensus_peers.cc:485] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5e4db5204e4acf8719100d02d533fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:43.974344 5105 tablet_bootstrap.cc:654] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.986748 6616 consensus_queue.cc:1035] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.989466 5103 tablet_bootstrap.cc:492] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:43.989856 5103 ts_tablet_manager.cc:1367] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.010s
I20240503 02:12:43.993463 5106 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:43.993759 5106 ts_tablet_manager.cc:1367] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20240503 02:12:43.994714 5106 raft_consensus.cc:348] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.995083 5106 raft_consensus.cc:374] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.995215 5106 raft_consensus.cc:729] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.995743 5106 consensus_queue.cc:260] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:43.996628 5106 ts_tablet_manager.cc:1398] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:43.997140 5106 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:43.998373 5114 ts_tablet_manager.cc:1888] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240503 02:12:43.999408 5106 tablet_bootstrap.cc:654] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.008303 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 9a8e996d76be4dda876f4645df56c1ce (table default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582]). Expected version 2 got 0
I20240503 02:12:44.009158 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:44.011428 5114 log.cc:1196] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:43.956856 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b9511e4de6c942b9b9e0790c88d3d543 (table default.Table_b417b8282cd14a9aa9bcffd17aab880c [id=e22df2e2673c45cbb1b32201a728333c]) successfully deleted
I20240503 02:12:44.010223 6560 consensus_queue.cc:1035] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:44.013149 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 56267us
I20240503 02:12:44.014055 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 87713e899d2148308316d807a6d76921 (table default.Table_4702ebe6617c4aebb1264298076c6a4d [id=31669972ad634736b38958103d7b6c16]) successfully deleted
I20240503 02:12:44.005321 5103 raft_consensus.cc:348] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.014264 5103 raft_consensus.cc:374] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.014369 5103 raft_consensus.cc:729] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.014910 5103 consensus_queue.cc:260] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.015177 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 70e08c10c15f41878202cef0c8d5ad3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e08c10c15f41878202cef0c8d5ad3c already in progress: creating tablet
I20240503 02:12:43.977077 5159 ts_tablet_manager.cc:1888] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.017290 5159 log.cc:1196] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:44.018360 5159 ts_tablet_manager.cc:1909] T 3f9994ca5aea48d3bc25c097dddba3c3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.018357 5103 ts_tablet_manager.cc:1398] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240503 02:12:44.018677 5103 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:44.020447 5103 tablet_bootstrap.cc:654] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.021718 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 858) took 2882 ms. Trace:
I20240503 02:12:44.016068 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c1f39e8d17b44895bd5c979d82145ad9 (table default.Table_9b099c5cc2d14c2ba0fad6df023e947c [id=ec6cee0c5c6c466682653bba626dc0c7]) successfully deleted
W20240503 02:12:44.021685 4170 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 55314us
I20240503 02:12:44.022116 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 79b8db679ba24ac6a93be591c0513ab0 (table default.Table_95c5dbfc7e2542dd93ea5839f66200e1 [id=ffbb2c8938d54556bf18d766e0c901cc]) successfully deleted
I20240503 02:12:44.023207 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 151f925dcb304594b60c85d850b83533 with delete_type TABLET_DATA_DELETED (replaced by tablet 7833167a80c34f3e85bf9e79185ad84a) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.023574 6620 consensus_queue.cc:1035] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.026116 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 151f925dcb304594b60c85d850b83533 with delete_type TABLET_DATA_DELETED (replaced by tablet 7833167a80c34f3e85bf9e79185ad84a) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.031201 5114 ts_tablet_manager.cc:1909] T c88eaba979394b449d96ee956c3f28a8 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.031687 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 849) took 2190 ms. Trace:
I20240503 02:12:44.031822 5114 rpcz_store.cc:276] 0503 02:12:41.841290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.841337 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:43.959518 (+2118181us) ts_tablet_manager.cc:1120] Deleting tablet c88eaba979394b449d96ee956c3f28a8
0503 02:12:43.959527 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.983768 (+ 24241us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.998248 (+ 14480us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.031656 (+ 33408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2078253}
I20240503 02:12:44.031962 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.035300 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.036608 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee32345fb9349f19af09a3447c4caad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.038069 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee32345fb9349f19af09a3447c4caad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.038678 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848c837598c49628c11374d3c875b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:44.040938 4462 consensus_peers.cc:485] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f862891b43a4845b723bfdb9a7da591. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.042038 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9709e189b5424f23a7663146d1a912ac" } modify_external_catalogs: true
I20240503 02:12:44.040973 5115 ts_tablet_manager.cc:1888] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.042279 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_9709e189b5424f23a7663146d1a912ac" } modify_external_catalogs: true
I20240503 02:12:44.042516 5115 log.cc:1196] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/41d44fe8f6334572996665bdec60e2e5
I20240503 02:12:44.042971 5115 ts_tablet_manager.cc:1909] T 41d44fe8f6334572996665bdec60e2e5 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:44.044198 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848c837598c49628c11374d3c875b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.044241 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.044438 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.048854 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0494ee4bd746069bfaf39bfb46687d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.032339 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 850) took 2188 ms. Trace:
I20240503 02:12:44.049124 5114 rpcz_store.cc:276] 0503 02:12:41.844165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.844215 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:44.032199 (+2187984us) ts_tablet_manager.cc:1120] Deleting tablet 7b86a6bb791449b5a4c490663d87afd4
0503 02:12:44.032211 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.032324 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150848}
I20240503 02:12:44.049726 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 759) took 4182 ms. Trace:
I20240503 02:12:44.022644 5105 tablet_bootstrap.cc:492] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.026895 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 151f925dcb304594b60c85d850b83533 with delete_type TABLET_DATA_DELETED (replaced by tablet 7833167a80c34f3e85bf9e79185ad84a) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.035622 6739 raft_consensus.cc:482] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.052488 5115 rpcz_store.cc:276] 0503 02:12:39.867706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.867950 (+ 244us) service_pool.cc:225] Handling call
0503 02:12:43.936694 (+4068744us) ts_tablet_manager.cc:1120] Deleting tablet 41d44fe8f6334572996665bdec60e2e5
0503 02:12:43.936707 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.001887 (+ 65180us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.040431 (+ 38544us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.049704 (+ 9273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4068556}
I20240503 02:12:44.051280 6698 raft_consensus.cc:482] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.052572 6739 raft_consensus.cc:504] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.054121 6698 raft_consensus.cc:504] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.054876 6739 leader_election.cc:288] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.055409 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.052675 5105 ts_tablet_manager.cc:1367] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.006s
I20240503 02:12:44.055693 6698 leader_election.cc:288] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:44.056042 4746 leader_election.cc:341] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.056200 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.056244 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9994ca5aea48d3bc25c097dddba3c3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.056447 4519 raft_consensus.cc:2454] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:44.056403 5105 raft_consensus.cc:348] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.056587 5105 raft_consensus.cc:374] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:44.056869 4608 leader_election.cc:341] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:44.057094 4608 leader_election.cc:302] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:44.057500 6699 raft_consensus.cc:2781] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.057590 6699 raft_consensus.cc:482] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.057664 6699 raft_consensus.cc:3037] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.036104 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet e82190dfb1b1418b9f20bf4d0c2c2409 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.037214 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee32345fb9349f19af09a3447c4caad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.042079 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848c837598c49628c11374d3c875b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:43 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.050246 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f862891b43a4845b723bfdb9a7da591 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.058810 5103 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:44.059012 5103 ts_tablet_manager.cc:1367] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
I20240503 02:12:44.059967 4519 raft_consensus.cc:3037] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.060467 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 851) took 2206 ms. Trace:
I20240503 02:12:44.060550 5114 rpcz_store.cc:276] 0503 02:12:41.853991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.854074 (+ 83us) service_pool.cc:225] Handling call
0503 02:12:44.060317 (+2206243us) ts_tablet_manager.cc:1120] Deleting tablet 7982ba6234b844f682cb23c0bfc4068c
0503 02:12:44.060327 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.060449 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2178915}
I20240503 02:12:44.060765 6699 raft_consensus.cc:504] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.061461 6699 leader_election.cc:288] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.053896 5106 tablet_bootstrap.cc:492] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.062292 5106 ts_tablet_manager.cc:1367] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240503 02:12:44.062705 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b8db679ba24ac6a93be591c0513ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.063516 5106 raft_consensus.cc:348] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.063809 5106 raft_consensus.cc:374] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.063963 5106 raft_consensus.cc:729] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.064365 5106 consensus_queue.cc:260] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.065574 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9994ca5aea48d3bc25c097dddba3c3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:44.065649 4746 leader_election.cc:341] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.065770 4746 leader_election.cc:302] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.065918 4519 raft_consensus.cc:3037] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.066253 5106 ts_tablet_manager.cc:1398] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:44.066896 5106 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.069185 4519 raft_consensus.cc:2454] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:44.066591 5115 tablet_replica.cc:330] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:44.071398 5115 raft_consensus.cc:2229] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.071995 5115 raft_consensus.cc:2258] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.072682 6739 raft_consensus.cc:2726] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:44.072706 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 706120be4d2a468780ffd64ea54849a3 consensus state has no leader
I20240503 02:12:44.072829 5115 ts_tablet_manager.cc:1875] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.073988 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_4511fe14eef348aa8f91987405ac7bfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.074661 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4511fe14eef348aa8f91987405ac7bfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.074724 4608 leader_election.cc:302] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.074673 6699 consensus_queue.cc:1035] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:44.075026 4608 leader_election.cc:341] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:44.075145 6699 raft_consensus.cc:2781] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.075379 6699 raft_consensus.cc:686] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:44.075562 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_74ec097848244b30973ad02961628793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.075655 6699 consensus_queue.cc:237] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.075991 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd4940fd3c16424eaaaef59ab50a04a8 consensus state has no leader
I20240503 02:12:44.076380 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3da2b86b13c8443c8c6156273fb9d097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.076989 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3da2b86b13c8443c8c6156273fb9d097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:44.078756 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ec097848244b30973ad02961628793 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.078608 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_103a11fe81534cdbb336f9b1b18bfb33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.079135 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_103a11fe81534cdbb336f9b1b18bfb33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.079293 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f10b8b1d68874a60a7900ec6fcc6e78c" } new_table_name: "default.Table_4a279b299cf046659678db65a1ee5ddc" modify_external_catalogs: true
I20240503 02:12:44.086145 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.052296 5159 rpcz_store.cc:276] 0503 02:12:41.139593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.139657 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:43.883006 (+2743349us) ts_tablet_manager.cc:1120] Deleting tablet 3f9994ca5aea48d3bc25c097dddba3c3
0503 02:12:43.883021 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:43.887160 (+ 4139us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:43.976950 (+ 89790us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.021692 (+ 44742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2741837}
I20240503 02:12:44.089720 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 860) took 2879 ms. Trace:
I20240503 02:12:44.089861 5159 rpcz_store.cc:276] 0503 02:12:41.210369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.210436 (+ 67us) service_pool.cc:225] Handling call
0503 02:12:44.089553 (+2879117us) ts_tablet_manager.cc:1120] Deleting tablet 0c3b9a888b894897b7cc3cf08402febd
0503 02:12:44.089567 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.089693 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2878933}
I20240503 02:12:44.090158 5159 tablet_replica.cc:330] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.090378 5159 raft_consensus.cc:2229] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.090597 5159 raft_consensus.cc:2258] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.090684 5106 tablet_bootstrap.cc:654] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.091468 5159 ts_tablet_manager.cc:1875] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.098474 6695 consensus_queue.cc:1035] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.056648 5105 raft_consensus.cc:729] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:44.098696 4746 consensus_peers.cc:485] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88eaba979394b449d96ee956c3f28a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.098896 5105 consensus_queue.cc:260] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.099890 5105 ts_tablet_manager.cc:1398] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.044s user 0.000s sys 0.002s
I20240503 02:12:44.100243 5105 tablet_bootstrap.cc:492] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.101012 5103 raft_consensus.cc:348] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.101244 5103 raft_consensus.cc:374] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.101301 5103 raft_consensus.cc:729] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.101817 5103 consensus_queue.cc:260] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.102461 5103 ts_tablet_manager.cc:1398] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20240503 02:12:44.103283 5105 tablet_bootstrap.cc:654] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:44.105032 4608 consensus_peers.cc:485] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd66f88a24b74f2c9b2050af081d4601. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.105540 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 852) took 2245 ms. Trace:
I20240503 02:12:44.105641 5114 rpcz_store.cc:276] 0503 02:12:41.860015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.860076 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:44.064906 (+2204830us) ts_tablet_manager.cc:1120] Deleting tablet ee16f1beb6a34a85bc872dedd64472ce
0503 02:12:44.105362 (+ 40456us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.105517 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183455}
I20240503 02:12:44.106338 6619 raft_consensus.cc:482] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.106622 6619 raft_consensus.cc:504] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.108245 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e49cde2fd4aab9f44921d3c3921a5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.108565 4667 raft_consensus.cc:2454] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:44.109220 5114 tablet_replica.cc:330] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:44.109459 4462 leader_election.cc:302] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.109884 6616 raft_consensus.cc:2781] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.109856 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 41d44fe8f6334572996665bdec60e2e5 (table default.Table_74b76179489146ee9c1af791b5976abe [id=d6973688027a4da2ba16399151d8bedd]) successfully deleted
I20240503 02:12:44.109961 6616 raft_consensus.cc:482] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.110020 6616 raft_consensus.cc:3037] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.110116 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]) successfully deleted
W20240503 02:12:44.110354 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7b86a6bb791449b5a4c490663d87afd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b86a6bb791449b5a4c490663d87afd4 already in progress: creating tablet
W20240503 02:12:44.110572 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7982ba6234b844f682cb23c0bfc4068c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7982ba6234b844f682cb23c0bfc4068c already in progress: creating tablet
I20240503 02:12:44.110651 6619 leader_election.cc:288] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:44.110766 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ee16f1beb6a34a85bc872dedd64472ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee16f1beb6a34a85bc872dedd64472ce already in progress: creating tablet
I20240503 02:12:44.109484 5114 raft_consensus.cc:2229] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.111258 5114 raft_consensus.cc:2258] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.114027 6616 raft_consensus.cc:504] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.114758 6616 leader_election.cc:288] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.114993 6758 tablet.cc:1722] T 9a8e996d76be4dda876f4645df56c1ce P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:44.115285 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e49cde2fd4aab9f44921d3c3921a5" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:44.115515 4667 raft_consensus.cc:3037] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.114048 5114 ts_tablet_manager.cc:1875] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.118366 4667 raft_consensus.cc:2454] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:44.119419 4462 leader_election.cc:302] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:44.119844 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2c283c08fe3453899834ff337798fc2 consensus state has no leader
W20240503 02:12:44.119992 4462 leader_election.cc:341] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 607e49cde2fd4aab9f44921d3c3921a5
W20240503 02:12:44.121382 4462 leader_election.cc:341] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 607e49cde2fd4aab9f44921d3c3921a5
I20240503 02:12:44.121526 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2
I20240503 02:12:44.121954 6756 tablet.cc:1722] T 9a8e996d76be4dda876f4645df56c1ce P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:44.122064 6619 raft_consensus.cc:2781] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.122156 6619 raft_consensus.cc:686] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:44.122207 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e725732066ff4ce69168444e3457b10c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.122470 6619 consensus_queue.cc:237] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.124794 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:44.125303 5106 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.125459 5106 ts_tablet_manager.cc:1367] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.001s
I20240503 02:12:44.126225 5106 raft_consensus.cc:348] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.130972 4266 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:44.114387 6757 tablet.cc:1722] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 02:12:44.127563 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75aa4b8ef74e4af18aa4390d593d665d consensus state has no leader
I20240503 02:12:44.132494 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet b9470480a9de45209044f8e8c461519a from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:44.133417 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_899f907d25f64238a341605f4dd1a34d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.129462 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 deleted and replaced by tablet 94a609dac84a4305a2cbf8af56f0c055
I20240503 02:12:44.134101 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a6d78e7a628d4b7b939e74d32d5ce601" } modify_external_catalogs: true
I20240503 02:12:44.134291 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a6d78e7a628d4b7b939e74d32d5ce601" } modify_external_catalogs: true
W20240503 02:12:44.135326 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_899f907d25f64238a341605f4dd1a34d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.136899 4266 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5904b0f9c29f483eab4730665a8bb2fd" } modify_external_catalogs: true
I20240503 02:12:44.137111 4266 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5904b0f9c29f483eab4730665a8bb2fd" } modify_external_catalogs: true
I20240503 02:12:44.137622 5105 tablet_bootstrap.cc:492] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.129987 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e0f6872c37c746fab51a59df01834e38" } modify_external_catalogs: true
I20240503 02:12:44.138226 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e0f6872c37c746fab51a59df01834e38" } modify_external_catalogs: true
W20240503 02:12:44.138657 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e725732066ff4ce69168444e3457b10c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.130496 4273 master_service.cc:946] ReplaceTablet: received request to replace tablet 9d900a0dc6c34076a0993210693a57a0 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:44.139654 5105 ts_tablet_manager.cc:1367] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240503 02:12:44.140318 5105 raft_consensus.cc:348] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.140533 5105 raft_consensus.cc:374] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.140589 5105 raft_consensus.cc:729] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.140776 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 73965590a0ea4bffae6e99fa1367fad0
I20240503 02:12:44.142083 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_950e46a8a847487e902ad641c4895f8c" } modify_external_catalogs: true
I20240503 02:12:44.142529 5105 consensus_queue.cc:260] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.142638 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_950e46a8a847487e902ad641c4895f8c" } modify_external_catalogs: true
I20240503 02:12:44.144256 6739 raft_consensus.cc:482] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.144402 6739 raft_consensus.cc:504] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.145056 6739 leader_election.cc:288] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.146194 5105 ts_tablet_manager.cc:1398] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:44.146494 5105 tablet_bootstrap.cc:492] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.147406 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e80cc31f5c4908b82f2b030255f6af" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.147858 5105 tablet_bootstrap.cc:654] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.149420 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3f9994ca5aea48d3bc25c097dddba3c3 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
W20240503 02:12:44.149688 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0c3b9a888b894897b7cc3cf08402febd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c3b9a888b894897b7cc3cf08402febd already in progress: creating tablet
I20240503 02:12:44.151532 5106 raft_consensus.cc:374] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.151721 5106 raft_consensus.cc:729] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.152109 6616 raft_consensus.cc:482] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.152022 5106 consensus_queue.cc:260] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.152413 6616 raft_consensus.cc:504] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.148123 4746 leader_election.cc:341] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.153012 5106 ts_tablet_manager.cc:1398] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240503 02:12:44.153179 6616 leader_election.cc:288] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.153348 5106 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.154165 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38d63d2cc8c4f54ba43cdf8cde1841f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.154425 4667 raft_consensus.cc:2454] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
W20240503 02:12:44.154592 4462 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:44.154798 4462 leader_election.cc:302] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:44.155084 6616 raft_consensus.cc:2781] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.155231 6616 raft_consensus.cc:482] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.155364 5106 tablet_bootstrap.cc:654] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.155400 6616 raft_consensus.cc:3037] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.151966 5159 ts_tablet_manager.cc:1888] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.156126 5159 log.cc:1196] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/27c13f257eae44689af07cc31ea254c4
I20240503 02:12:44.158457 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 94a609dac84a4305a2cbf8af56f0c055) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.159129 6616 raft_consensus.cc:504] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.159291 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 035216c9475d4b59aa1ac81b285b4d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.159862 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 94a609dac84a4305a2cbf8af56f0c055) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.160392 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38d63d2cc8c4f54ba43cdf8cde1841f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:44.160730 4667 raft_consensus.cc:3037] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.162027 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.162153 5159 ts_tablet_manager.cc:1909] T 27c13f257eae44689af07cc31ea254c4 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.162752 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.163439 4667 raft_consensus.cc:2454] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:44.164059 6762 consensus_queue.cc:1035] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.164171 6739 raft_consensus.cc:482] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.164412 6739 raft_consensus.cc:504] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.164635 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 862) took 2952 ms. Trace:
I20240503 02:12:44.164716 5159 rpcz_store.cc:276] 0503 02:12:41.211756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.212779 (+ 1023us) service_pool.cc:225] Handling call
0503 02:12:44.090109 (+2877330us) ts_tablet_manager.cc:1120] Deleting tablet 27c13f257eae44689af07cc31ea254c4
0503 02:12:44.090119 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.130430 (+ 40311us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.151879 (+ 21449us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.164614 (+ 12735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2877145}
I20240503 02:12:44.165086 6739 leader_election.cc:288] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.167600 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84856917cb84c2a853c9a8a80246704" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:44.168243 4746 leader_election.cc:341] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.161727 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9994ca5aea48d3bc25c097dddba3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.161943 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 73965590a0ea4bffae6e99fa1367fad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:44.160799 4462 leader_election.cc:341] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:44.168584 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84856917cb84c2a853c9a8a80246704" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.170257 4462 leader_election.cc:302] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
W20240503 02:12:44.170507 4746 leader_election.cc:341] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.170644 4746 leader_election.cc:302] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.170931 6620 raft_consensus.cc:2781] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.171057 6620 raft_consensus.cc:686] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:44.171324 6739 raft_consensus.cc:2726] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.171330 6620 consensus_queue.cc:237] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.160012 6616 leader_election.cc:288] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:44.179854 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96a655b2dbdd4b9091584c6cf8a8d907 has no consensus state
I20240503 02:12:44.182119 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet b9470480a9de45209044f8e8c461519a
I20240503 02:12:44.182871 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 8d1ac81e1a6f41cf974bb30e6b74eea2
I20240503 02:12:44.184756 5268 master_runner.cc:186] Error getting master registration for 127.4.17.124:46689: OK, instance_id { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" instance_seqno: 1714702356513402 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:12:44.185107 5268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:12:44.185249 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9d900a0dc6c34076a0993210693a57a0
I20240503 02:12:44.185700 4273 catalog_manager.cc:6554] ReplaceTablet: tablet 9d900a0dc6c34076a0993210693a57a0 deleted and replaced by tablet 246fa4d9be0a4d6da4a1b26a188290c8
I20240503 02:12:44.186044 4274 catalog_manager.cc:6554] ReplaceTablet: tablet b9470480a9de45209044f8e8c461519a deleted and replaced by tablet 620223977a5f48209f6cc12b53915508
I20240503 02:12:44.186242 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7c19263f06ec41d7a6fe6a8df7f88f8f
I20240503 02:12:44.187381 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 89f3232d509f4cb39eb8923fe1d120a7
I20240503 02:12:44.195286 5103 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:44.197988 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet b84856917cb84c2a853c9a8a80246704 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.198822 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet b9470480a9de45209044f8e8c461519a with delete_type TABLET_DATA_DELETED (replaced by tablet 620223977a5f48209f6cc12b53915508) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.198891 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet b9470480a9de45209044f8e8c461519a with delete_type TABLET_DATA_DELETED (replaced by tablet 620223977a5f48209f6cc12b53915508) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.199438 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet b9470480a9de45209044f8e8c461519a with delete_type TABLET_DATA_DELETED (replaced by tablet 620223977a5f48209f6cc12b53915508) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.200003 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.200071 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.201262 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.145548 5115 ts_tablet_manager.cc:1888] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.148834 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e80cc31f5c4908b82f2b030255f6af" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.150317 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 94a609dac84a4305a2cbf8af56f0c055) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.201547 6739 raft_consensus.cc:482] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.201691 6739 raft_consensus.cc:504] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.202055 4746 leader_election.cc:341] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.202150 4746 leader_election.cc:302] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.202364 6738 raft_consensus.cc:2726] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.202579 6739 leader_election.cc:288] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.203263 5115 log.cc:1196] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/f146f7cce0b24d3087bfe0973e2b7ef0
I20240503 02:12:44.203270 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73ec6543b9a468492f3b97772ce43f9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.203760 5115 ts_tablet_manager.cc:1909] T f146f7cce0b24d3087bfe0973e2b7ef0 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:44.203859 4746 leader_election.cc:341] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.204530 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d900a0dc6c34076a0993210693a57a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 246fa4d9be0a4d6da4a1b26a188290c8) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.205119 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d900a0dc6c34076a0993210693a57a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 246fa4d9be0a4d6da4a1b26a188290c8) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.205294 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73ec6543b9a468492f3b97772ce43f9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.205849 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d900a0dc6c34076a0993210693a57a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 246fa4d9be0a4d6da4a1b26a188290c8) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:44.205978 4746 leader_election.cc:341] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.206080 4746 leader_election.cc:302] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.206543 6739 raft_consensus.cc:2726] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.209146 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.209492 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.209527 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:44.210052 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.210729 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.211298 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.215538 4275 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_7e85ce8838ff4bb8a5551abe22389f43 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 2 got 0
I20240503 02:12:44.217839 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f3232d509f4cb39eb8923fe1d120a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.223278 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 760) took 4355 ms. Trace:
I20240503 02:12:44.223565 5115 rpcz_store.cc:276] 0503 02:12:39.867777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.868162 (+ 385us) service_pool.cc:225] Handling call
0503 02:12:44.066491 (+4198329us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:44.066502 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.142109 (+ 75607us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.145380 (+ 3271us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.223227 (+ 77847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4198191}
I20240503 02:12:44.224058 5115 tablet_replica.cc:330] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:44.224345 5115 raft_consensus.cc:2229] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.224624 5115 raft_consensus.cc:2258] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.225335 5115 ts_tablet_manager.cc:1875] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.225462 5103 tablet_bootstrap.cc:654] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.227792 5159 tablet_replica.cc:330] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.228106 5159 raft_consensus.cc:2229] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.229274 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 27c13f257eae44689af07cc31ea254c4 (table default.Table_adc5b950ff6b4d9e8f02719738c475c5 [id=ce02819e30324e95aa0ae9fadf8836f1]) successfully deleted
I20240503 02:12:44.229653 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet f146f7cce0b24d3087bfe0973e2b7ef0 (table default.Table_18dc31f244e94d6ab3c576eaae5e408b [id=64f4026243214e1bbed391c6e0e7df76]) successfully deleted
I20240503 02:12:44.230440 5159 raft_consensus.cc:2258] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.231102 5159 ts_tablet_manager.cc:1875] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.233358 6762 raft_consensus.cc:482] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.233566 6762 raft_consensus.cc:504] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.234302 6762 leader_election.cc:288] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.234313 6732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:44.234630 6732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:44.237506 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df63329dc2b149d9b08cea341b540d1d. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:44.237844 5105 tablet_bootstrap.cc:492] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.237893 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:12:44.238005 5105 ts_tablet_manager.cc:1367] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.001s
I20240503 02:12:44.241072 5105 raft_consensus.cc:348] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.241472 5105 raft_consensus.cc:374] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.241592 5105 raft_consensus.cc:729] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.241919 5105 consensus_queue.cc:260] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.242682 5105 ts_tablet_manager.cc:1398] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:44.243033 5105 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.244510 5105 tablet_bootstrap.cc:654] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.245640 6732 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:44.247296 5159 ts_tablet_manager.cc:1888] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.247503 5159 log.cc:1196] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:44.247912 5159 ts_tablet_manager.cc:1909] T 20ce11c70a924b8d92d8f36fc7401dfd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.249176 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 861) took 3037 ms. Trace:
I20240503 02:12:44.250274 5159 rpcz_store.cc:276] 0503 02:12:41.211630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.213457 (+ 1827us) service_pool.cc:225] Handling call
0503 02:12:44.227736 (+3014279us) ts_tablet_manager.cc:1120] Deleting tablet 20ce11c70a924b8d92d8f36fc7401dfd
0503 02:12:44.227746 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.234313 (+ 6567us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.247175 (+ 12862us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.249139 (+ 1964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014136}
I20240503 02:12:44.251552 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 863) took 3013 ms. Trace:
I20240503 02:12:44.251917 5159 rpcz_store.cc:276] 0503 02:12:41.237895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.237969 (+ 74us) service_pool.cc:225] Handling call
0503 02:12:44.251394 (+3013425us) ts_tablet_manager.cc:1120] Deleting tablet 5f8656a886564f06931039b516162b51
0503 02:12:44.251406 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.251530 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3000141}
I20240503 02:12:44.252843 5105 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.253347 5105 ts_tablet_manager.cc:1367] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:12:44.254209 5105 raft_consensus.cc:348] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.254496 5105 raft_consensus.cc:374] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.254611 5105 raft_consensus.cc:729] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.254974 5105 consensus_queue.cc:260] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.255715 5105 ts_tablet_manager.cc:1398] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:44.256127 5105 tablet_bootstrap.cc:492] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.256773 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f39e8d17b44895bd5c979d82145ad9" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:44.257758 4608 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.258157 5105 tablet_bootstrap.cc:654] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:44.259073 4608 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:44.259227 4608 leader_election.cc:302] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.259827 6695 raft_consensus.cc:2726] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.265267 6739 raft_consensus.cc:482] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.265494 6739 raft_consensus.cc:504] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.266379 6739 leader_election.cc:288] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.267879 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32c377006a24372859ed44d59cf90ee" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:44.268709 4746 leader_election.cc:341] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.268867 5106 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.269084 5106 ts_tablet_manager.cc:1367] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.003s
I20240503 02:12:44.269325 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32c377006a24372859ed44d59cf90ee" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:44.270298 4746 leader_election.cc:341] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.270471 4746 leader_election.cc:302] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.270826 5106 raft_consensus.cc:348] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.271018 6739 raft_consensus.cc:2726] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.271286 5106 raft_consensus.cc:374] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.271432 5106 raft_consensus.cc:729] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.271850 5106 consensus_queue.cc:260] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.272646 5106 ts_tablet_manager.cc:1398] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:44.275617 5106 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.278555 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 864) took 3040 ms. Trace:
I20240503 02:12:44.278757 5159 rpcz_store.cc:276] 0503 02:12:41.238489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.238578 (+ 89us) service_pool.cc:225] Handling call
0503 02:12:44.278392 (+3039814us) ts_tablet_manager.cc:1120] Deleting tablet 28c1a81589df496b92b980353f00e594
0503 02:12:44.278403 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.278526 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027066}
I20240503 02:12:44.279253 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 865) took 3040 ms. Trace:
I20240503 02:12:44.279376 5159 rpcz_store.cc:276] 0503 02:12:41.238910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.238963 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:44.279129 (+3040166us) ts_tablet_manager.cc:1120] Deleting tablet efd8e0353b5f4ce484ac81d646b5323d
0503 02:12:44.279140 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.279235 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027761}
I20240503 02:12:44.279806 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 866) took 3040 ms. Trace:
I20240503 02:12:44.279944 5159 rpcz_store.cc:276] 0503 02:12:41.239654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.239704 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:44.279709 (+3040005us) ts_tablet_manager.cc:1120] Deleting tablet 39c48b14bc624c28a871d45e9429ee30
0503 02:12:44.279719 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.279792 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028295}
I20240503 02:12:44.280308 5106 tablet_bootstrap.cc:654] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.280542 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 867) took 3039 ms. Trace:
I20240503 02:12:44.280692 5159 rpcz_store.cc:276] 0503 02:12:41.240613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.240665 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:44.280409 (+3039744us) ts_tablet_manager.cc:1120] Deleting tablet 3c421694ce1e424da7953d061a6bca3e
0503 02:12:44.280420 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.280522 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028943}
I20240503 02:12:44.281221 5159 tablet_replica.cc:330] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.281632 5159 raft_consensus.cc:2229] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.281873 5159 raft_consensus.cc:2258] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.282696 5159 ts_tablet_manager.cc:1875] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.283627 6739 raft_consensus.cc:482] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.283769 6739 raft_consensus.cc:504] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.284425 6739 leader_election.cc:288] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.285526 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.285813 4667 raft_consensus.cc:2454] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:44.286350 4746 leader_election.cc:302] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:44.287055 6739 raft_consensus.cc:2781] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.287135 6739 raft_consensus.cc:482] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.287192 6739 raft_consensus.cc:3037] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.289889 5106 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.290058 5106 ts_tablet_manager.cc:1367] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 02:12:44.290282 5114 ts_tablet_manager.cc:1888] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.290416 5114 log.cc:1196] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/66ca08877cd948e1b04736b54bc779db
I20240503 02:12:44.289947 6739 raft_consensus.cc:504] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.291994 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:44.290957 5106 raft_consensus.cc:348] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.292763 4746 leader_election.cc:341] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.293071 5106 raft_consensus.cc:374] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.293124 6739 leader_election.cc:288] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.293399 5106 raft_consensus.cc:729] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.293881 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:44.294114 4667 raft_consensus.cc:3037] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.293984 5106 consensus_queue.cc:260] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.295192 5106 ts_tablet_manager.cc:1398] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:44.295519 5106 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.296846 4667 raft_consensus.cc:2454] T c2efdbfa909c475d95e99cf24206608f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:44.297591 4746 leader_election.cc:302] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.298203 6739 raft_consensus.cc:2781] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.298509 6739 raft_consensus.cc:686] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:44.298843 6739 consensus_queue.cc:237] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.299155 5106 tablet_bootstrap.cc:654] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.300338 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.300704 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 20ce11c70a924b8d92d8f36fc7401dfd (table default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]) successfully deleted
W20240503 02:12:44.301090 4746 leader_election.cc:341] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:44.301218 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5f8656a886564f06931039b516162b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8656a886564f06931039b516162b51 already in progress: creating tablet
W20240503 02:12:44.301465 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 28c1a81589df496b92b980353f00e594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28c1a81589df496b92b980353f00e594 already in progress: creating tablet
W20240503 02:12:44.301676 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet efd8e0353b5f4ce484ac81d646b5323d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd8e0353b5f4ce484ac81d646b5323d already in progress: creating tablet
W20240503 02:12:44.301910 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 39c48b14bc624c28a871d45e9429ee30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39c48b14bc624c28a871d45e9429ee30 already in progress: creating tablet
W20240503 02:12:44.302106 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3c421694ce1e424da7953d061a6bca3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c421694ce1e424da7953d061a6bca3e already in progress: creating tablet
I20240503 02:12:44.303896 4275 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_9e181c295fb749eaa9445ce83763c3e8 [id=12382a23327a42f8bf299f03fa08b582] alter complete (version 2)
I20240503 02:12:44.304885 4574 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:44.307472 6774 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 02:12:44.314204 4720 heartbeater.cc:347] Connected to a master server at 127.4.17.124:46689
I20240503 02:12:44.314786 5106 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.315621 5106 ts_tablet_manager.cc:1367] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20240503 02:12:44.317245 5106 raft_consensus.cc:348] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.318017 5106 raft_consensus.cc:374] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.318302 5106 raft_consensus.cc:729] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.319473 5106 consensus_queue.cc:260] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.323804 5106 ts_tablet_manager.cc:1398] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:44.325129 5106 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.326464 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_80c858c6cd0b407da24b467f885e2b76" } new_table_name: "default.Table_8445a7ddbfe94d78a5489dd954b51d8b" modify_external_catalogs: true
I20240503 02:12:44.329030 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_131368ef1ddb4611aeb046287d0ac6b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.329711 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_131368ef1ddb4611aeb046287d0ac6b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.331580 6695 raft_consensus.cc:482] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.335188 4275 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_7e85ce8838ff4bb8a5551abe22389f43" } new_table_name: "default.Table_8c5ce4d70c734f70a52c3e3267097103" modify_external_catalogs: true
I20240503 02:12:44.336309 6699 raft_consensus.cc:482] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.336470 6699 raft_consensus.cc:504] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.337344 6699 leader_election.cc:288] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:44.338323 4608 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:44.349045 6695 raft_consensus.cc:504] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.349647 5159 ts_tablet_manager.cc:1888] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.349869 5159 log.cc:1196] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/53b149e66a0446eeade190dd7f5d2bec
I20240503 02:12:44.350678 5106 tablet_bootstrap.cc:654] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.352301 5159 ts_tablet_manager.cc:1909] T 53b149e66a0446eeade190dd7f5d2bec P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.352989 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 892) took 2452 ms. Trace:
I20240503 02:12:44.354279 6695 leader_election.cc:288] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.354427 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 881 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:44.353822 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:44.357311 4813 raft_consensus.cc:2454] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:44.357712 5159 rpcz_store.cc:276] 0503 02:12:41.900504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.900712 (+ 208us) service_pool.cc:225] Handling call
0503 02:12:44.281169 (+2380457us) ts_tablet_manager.cc:1120] Deleting tablet 53b149e66a0446eeade190dd7f5d2bec
0503 02:12:44.281181 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.312048 (+ 30867us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.347486 (+ 35438us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.352959 (+ 5473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2378123}
I20240503 02:12:44.357829 4608 leader_election.cc:302] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:44.358201 6695 raft_consensus.cc:2781] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.358356 6695 raft_consensus.cc:482] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.358494 6695 raft_consensus.cc:3037] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.359695 5159 tablet_replica.cc:330] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.359905 5159 raft_consensus.cc:2229] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.360088 5159 raft_consensus.cc:2258] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.360800 4266 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1b13ea83ffa54dcbbfebaaa76e1686a9
I20240503 02:12:44.361289 4266 catalog_manager.cc:6554] ReplaceTablet: tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 deleted and replaced by tablet 5a4436ab9e7542f5b3aca930f6c5a758
I20240503 02:12:44.361961 6695 raft_consensus.cc:504] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.362349 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_7dbc61340b924e26bc747d93095af77b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.362573 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cd9386a1ac724110944b2963851c7442"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.363030 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd9386a1ac724110944b2963851c7442 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.363828 5159 ts_tablet_manager.cc:1875] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.364410 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6815736795654b079117f0285ae63c1c" } new_table_name: "default.Table_c8119b35d09a431f9eb89fa4fbfbe4f1" modify_external_catalogs: true
I20240503 02:12:44.362529 5106 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.365202 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_aca5c69ca17445739ee6c15a01908c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.365381 5106 ts_tablet_manager.cc:1367] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
W20240503 02:12:44.365870 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aca5c69ca17445739ee6c15a01908c4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.366078 5106 raft_consensus.cc:348] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.366277 5106 raft_consensus.cc:374] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.366330 5106 raft_consensus.cc:729] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.366570 5106 consensus_queue.cc:260] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.367204 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_cf194cb302824ebe8a359b8225c7725a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.367686 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf194cb302824ebe8a359b8225c7725a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.362344 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d3383012c69d498887d4cb8f518bcdd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.369596 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3383012c69d498887d4cb8f518bcdd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.369796 5106 ts_tablet_manager.cc:1398] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:44.370129 5106 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.371634 5106 tablet_bootstrap.cc:654] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.373371 6238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35434 (request call id 0) took 7109 ms. Trace:
I20240503 02:12:44.375931 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5be4b47f18164bf1882c0d2e917e43ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.376160 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:44.376400 4813 raft_consensus.cc:3037] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:44.376844 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5be4b47f18164bf1882c0d2e917e43ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:44.363036 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dbc61340b924e26bc747d93095af77b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.379364 4813 raft_consensus.cc:2454] T 1975c790bfa4449baa6d7b81509c5003 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:44.365872 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_eee9e761694041d69f22b08ae4363113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.380452 4608 leader_election.cc:302] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:44.383175 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eee9e761694041d69f22b08ae4363113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.383311 6699 raft_consensus.cc:2781] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.383437 6699 raft_consensus.cc:686] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:44.383661 6699 consensus_queue.cc:237] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.384495 5159 ts_tablet_manager.cc:1888] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.384608 5159 log.cc:1196] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8195767d80384a6c9a4690292f6776de
I20240503 02:12:44.387259 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a4436ab9e7542f5b3aca930f6c5a758) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.387917 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 53b149e66a0446eeade190dd7f5d2bec (table default.Table_dfe5a5a3b0c2426c912738f842cd96bf [id=117dbc7727ae404e87952f3088016923]) successfully deleted
I20240503 02:12:44.390470 4268 catalog_manager.cc:5484] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.390967 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:44.391471 4268 catalog_manager.cc:5484] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.391849 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 289300b955dd41c09d2fc348224bb9f4 (table default.Table_b145550ee4dc4b6a8f00b076f2ac72a5 [id=2817ff73927a4593bddef5408db5cc49]). Expected version 1 got 0
I20240503 02:12:44.392660 6738 raft_consensus.cc:482] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.392845 6738 raft_consensus.cc:504] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.393543 6738 leader_election.cc:288] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.394166 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a4436ab9e7542f5b3aca930f6c5a758) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.399860 6738 raft_consensus.cc:482] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.400043 6738 raft_consensus.cc:504] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.400686 6738 leader_election.cc:288] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.402348 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a4436ab9e7542f5b3aca930f6c5a758) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.402592 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c757a3a979c2489daa621cf8ea293ea8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:44.403456 4746 leader_election.cc:341] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.405094 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1fac803dd84eee8722f0a14559faf6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:44.405833 4746 leader_election.cc:341] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.406539 5115 ts_tablet_manager.cc:1888] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:44.406723 5115 log.cc:1196] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/65774b69a8bd4664a6b55b84eba37f7f
I20240503 02:12:44.407122 5115 ts_tablet_manager.cc:1909] T 65774b69a8bd4664a6b55b84eba37f7f P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:44.407516 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 758) took 4539 ms. Trace:
I20240503 02:12:44.407665 5115 rpcz_store.cc:276] 0503 02:12:39.867603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.868237 (+ 634us) service_pool.cc:225] Handling call
0503 02:12:44.223989 (+4355752us) ts_tablet_manager.cc:1120] Deleting tablet 65774b69a8bd4664a6b55b84eba37f7f
0503 02:12:44.223999 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.228507 (+ 4508us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.406387 (+177880us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.407495 (+ 1108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355591}
I20240503 02:12:44.408175 5115 tablet_replica.cc:330] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:44.408444 5115 raft_consensus.cc:2229] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.410717 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 65774b69a8bd4664a6b55b84eba37f7f (table default.Table_5df26c990f414242ac526c32679385fe [id=522166873101443c80e2c2159f181c71]) successfully deleted
I20240503 02:12:44.410943 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 83ff682e44214c4c809ac4a12b2f23d2 (DEFAULT_TABLE table=default.Table_fc35d19d5cd54bc2824834be14a6bb14 [id=b077c9cd1617497bbbca20af7aed5f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.411412 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ff682e44214c4c809ac4a12b2f23d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.413208 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 939e2b3230e645e881a8f926efae550e (DEFAULT_TABLE table=default.Table_2785fa72fa1e42629f30b99a8dc42925 [id=5cb67145cf554e539e98fc4ded1ebe5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.413589 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939e2b3230e645e881a8f926efae550e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.399652 6226 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "acc6d33969e74c209310005ae911c405" instance_seqno: 1714702335045023) as {username='slave'} at 127.4.17.67:45417; Asking this server to re-register.
I20240503 02:12:44.418979 4858 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:44.419332 4781 tablet_service.cc:1467] Processing CreateTablet for tablet d4b24a9822cd4e3693cf7511c2dc2eb9 (DEFAULT_TABLE table=default.Table_2e9fd306667548278723f880bb82a806 [id=a37ff45de8d545f38ea07539538f7eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.420135 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b24a9822cd4e3693cf7511c2dc2eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.428819 6757 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 02:12:44.437255 5159 ts_tablet_manager.cc:1909] T 8195767d80384a6c9a4690292f6776de P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.438920 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 83ff682e44214c4c809ac4a12b2f23d2 (DEFAULT_TABLE table=default.Table_fc35d19d5cd54bc2824834be14a6bb14 [id=b077c9cd1617497bbbca20af7aed5f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.439407 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 939e2b3230e645e881a8f926efae550e (DEFAULT_TABLE table=default.Table_2785fa72fa1e42629f30b99a8dc42925 [id=5cb67145cf554e539e98fc4ded1ebe5f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:44.439692 4462 consensus_peers.cc:485] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b4972057fae45168756d7c28ddbe2bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.439879 4497 tablet_service.cc:1467] Processing CreateTablet for tablet d4b24a9822cd4e3693cf7511c2dc2eb9 (DEFAULT_TABLE table=default.Table_2e9fd306667548278723f880bb82a806 [id=a37ff45de8d545f38ea07539538f7eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.440332 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c757a3a979c2489daa621cf8ea293ea8" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.440789 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1fac803dd84eee8722f0a14559faf6" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:44.441538 4746 leader_election.cc:341] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.441680 4746 leader_election.cc:302] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.443060 6738 raft_consensus.cc:2726] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.444046 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7219ce14a9ae4488b97914b9d02995f9
I20240503 02:12:44.447088 4271 catalog_manager.cc:5484] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
W20240503 02:12:44.450074 4746 leader_election.cc:341] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.450230 4746 leader_election.cc:302] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.450673 6738 raft_consensus.cc:2726] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.451859 5103 tablet_bootstrap.cc:492] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:44.454561 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b32c377006a24372859ed44d59cf90ee (table default.Table_80dedb822df54ccc9e63c27319fec3b9 [id=921e31bf406040b9a44409fdfbb79d2e]). Expected version 2 got 0
I20240503 02:12:44.456032 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f10e78f12a3948bca598aef0a06d8e58" } modify_external_catalogs: true
I20240503 02:12:44.456281 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_f10e78f12a3948bca598aef0a06d8e58" } modify_external_catalogs: true
I20240503 02:12:44.457739 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_94b433051bd4452cbb4566bba7d627a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.439806 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
W20240503 02:12:44.458366 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94b433051bd4452cbb4566bba7d627a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:44.458384 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 118635us
I20240503 02:12:44.459703 4528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:44.460659 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ff682e44214c4c809ac4a12b2f23d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.460970 4268 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4b3d4e5854b34468a48715e9093f70e2" } modify_external_catalogs: true
I20240503 02:12:44.461134 4268 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4b3d4e5854b34468a48715e9093f70e2" } modify_external_catalogs: true
I20240503 02:12:44.451066 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_55379d2e66cd42d28abf6943d13ca28e" } new_table_name: "default.Table_731eebc7b98e4b78832e9d33d32e0b5e" modify_external_catalogs: true
I20240503 02:12:44.451843 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eae0f8da0ab0497e8731eb30c28c8c7c" } modify_external_catalogs: true
I20240503 02:12:44.463554 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eae0f8da0ab0497e8731eb30c28c8c7c" } modify_external_catalogs: true
I20240503 02:12:44.452535 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f9c98f7833384e2cbb77a5a16f0bc77c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.465108 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9c98f7833384e2cbb77a5a16f0bc77c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.453284 4274 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_82219dec0a7945719fe7415d05e92a13" } new_table_name: "default.Table_b940744d9b8c4621a7edec7535949353" modify_external_catalogs: true
W20240503 02:12:44.453756 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77fa5be5a4f64c189ef619bc38e21eae consensus state has no leader
I20240503 02:12:44.467460 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_055c133ec2d14ff7b5940ff3d79ad420" } modify_external_catalogs: true
I20240503 02:12:44.467680 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_055c133ec2d14ff7b5940ff3d79ad420" } modify_external_catalogs: true
I20240503 02:12:44.467860 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b24a9822cd4e3693cf7511c2dc2eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.468858 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939e2b3230e645e881a8f926efae550e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.481696 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce11c70a924b8d92d8f36fc7401dfd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.482625 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1975c790bfa4449baa6d7b81509c5003" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.482887 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.482978 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.483062 5105 tablet_bootstrap.cc:492] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.483214 5105 ts_tablet_manager.cc:1367] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.227s user 0.024s sys 0.032s
I20240503 02:12:44.483899 5105 raft_consensus.cc:348] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.484099 5105 raft_consensus.cc:374] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.484162 5105 raft_consensus.cc:729] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.484378 5105 consensus_queue.cc:260] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.485826 5103 ts_tablet_manager.cc:1367] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.258s user 0.004s sys 0.000s
I20240503 02:12:44.486553 5103 raft_consensus.cc:348] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.486840 5103 raft_consensus.cc:374] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.486987 5103 raft_consensus.cc:729] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.487311 5103 consensus_queue.cc:260] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.488209 5103 ts_tablet_manager.cc:1398] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:44.487345 4494 tablet_service.cc:1467] Processing CreateTablet for tablet f14b1c210cf042838016de53574b0ef4 (DEFAULT_TABLE table=default.Table_2c4258a98cf14b69af1ffe992cf712e2 [id=e5b56ec392f74745bb748ef178755789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.488699 5114 ts_tablet_manager.cc:1909] T 66ca08877cd948e1b04736b54bc779db P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.489252 5103 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:44.491303 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 (DEFAULT_TABLE table=default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6 [id=0f6b37021cc04323af6f820204114af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.493341 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 61842978046d4cf5a4b123b9052bc754 (DEFAULT_TABLE table=default.Table_15995c9366f647308e1abed005edeb79 [id=e68f9d54345643248131db5a41806ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.496311 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 6d578b8abd804e19a45d634721760ff7 (DEFAULT_TABLE table=default.Table_9df571d7b21a4cce8f3f5423c9d50a37 [id=38262bbd9346419e853465e448fbafe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.484863 5105 ts_tablet_manager.cc:1398] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:44.490092 4788 tablet_service.cc:1467] Processing CreateTablet for tablet f14b1c210cf042838016de53574b0ef4 (DEFAULT_TABLE table=default.Table_2c4258a98cf14b69af1ffe992cf712e2 [id=e5b56ec392f74745bb748ef178755789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.492365 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 (DEFAULT_TABLE table=default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6 [id=0f6b37021cc04323af6f820204114af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.494714 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 61842978046d4cf5a4b123b9052bc754 (DEFAULT_TABLE table=default.Table_15995c9366f647308e1abed005edeb79 [id=e68f9d54345643248131db5a41806ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.497524 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 6d578b8abd804e19a45d634721760ff7 (DEFAULT_TABLE table=default.Table_9df571d7b21a4cce8f3f5423c9d50a37 [id=38262bbd9346419e853465e448fbafe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.498031 4493 tablet_service.cc:1467] Processing CreateTablet for tablet cced6c58c1554d1dad38a0f553a430e1 (DEFAULT_TABLE table=default.Table_84073519a2f04b50863dc1ea8bac73b9 [id=dc351d99b5e54398adecc427a44cbfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.499207 4785 tablet_service.cc:1467] Processing CreateTablet for tablet cced6c58c1554d1dad38a0f553a430e1 (DEFAULT_TABLE table=default.Table_84073519a2f04b50863dc1ea8bac73b9 [id=dc351d99b5e54398adecc427a44cbfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.499402 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 96a655b2dbdd4b9091584c6cf8a8d907 (DEFAULT_TABLE table=default.Table_4a279b299cf046659678db65a1ee5ddc [id=c60587696f8f4d3cb1d74085fd8de267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.499764 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 96a655b2dbdd4b9091584c6cf8a8d907 (DEFAULT_TABLE table=default.Table_4a279b299cf046659678db65a1ee5ddc [id=c60587696f8f4d3cb1d74085fd8de267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.500624 4499 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7b745bb994555a3e513f92ff29429 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.501567 4777 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7b745bb994555a3e513f92ff29429 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.502540 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 93cc893b1f664fddbb2833cb0b9843bd (DEFAULT_TABLE table=default.Table_4b0e3fb2ecb24f7ea287a2cbabe12de8 [id=2e843e566e6941e5aedeeceed7c1da8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.503574 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 40813cdfee114fc2ac10c45130b11f98 (DEFAULT_TABLE table=default.Table_0f61ff50b4434dbbaae6066ce10e5f6b [id=127970bcecef4f299992bd8203ddbe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.508476 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81213d7b9fc9429abd17f93b9f6982c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.508777 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a655b2dbdd4b9091584c6cf8a8d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.509136 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7b745bb994555a3e513f92ff29429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.513090 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cced6c58c1554d1dad38a0f553a430e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.516309 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 855) took 2447 ms. Trace:
I20240503 02:12:44.516453 5114 rpcz_store.cc:276] 0503 02:12:42.069102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.069161 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:44.106045 (+2036884us) ts_tablet_manager.cc:1120] Deleting tablet 66ca08877cd948e1b04736b54bc779db
0503 02:12:44.109158 (+ 3113us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.200522 (+ 91364us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.290116 (+ 89594us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.516257 (+226141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2036685}
I20240503 02:12:44.516855 5114 tablet_replica.cc:330] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:44.517210 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 9003cc376aa149b59947bc32bd071163
I20240503 02:12:44.517895 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b1c210cf042838016de53574b0ef4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:44.517987 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a4436ab9e7542f5b3aca930f6c5a758 has no consensus state
I20240503 02:12:44.518402 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet f40706400f114def9e14e6d1d7114962
W20240503 02:12:44.519465 4274 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70506e39cd742ed81ac6c7133ae5095 consensus state has no leader
I20240503 02:12:44.519860 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7f88d274b7654ba4942e757ba3302101
I20240503 02:12:44.520519 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet bd4940fd3c16424eaaaef59ab50a04a8
I20240503 02:12:44.521126 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d578b8abd804e19a45d634721760ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.522920 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88d274b7654ba4942e757ba3302101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.523519 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.524736 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 9003cc376aa149b59947bc32bd071163 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.524989 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet f40706400f114def9e14e6d1d7114962 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.525543 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61842978046d4cf5a4b123b9052bc754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.531175 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.534507 5114 raft_consensus.cc:2229] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.534971 5114 raft_consensus.cc:2258] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.535226 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 66ca08877cd948e1b04736b54bc779db (table default.Table_e8ca78c1991543479153a973a8346732 [id=997231149d3e427f824ed751b345d7b8]) successfully deleted
I20240503 02:12:44.535609 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet f40706400f114def9e14e6d1d7114962 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.535840 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88d274b7654ba4942e757ba3302101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.536001 4503 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.540221 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a655b2dbdd4b9091584c6cf8a8d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.545181 5105 tablet_bootstrap.cc:492] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.545838 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 893) took 2644 ms. Trace:
I20240503 02:12:44.545940 5159 rpcz_store.cc:276] 0503 02:12:41.901428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.901659 (+ 231us) service_pool.cc:225] Handling call
0503 02:12:44.359639 (+2457980us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:44.359651 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.380978 (+ 21327us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.384387 (+ 3409us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.545812 (+161425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456511}
I20240503 02:12:44.546344 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 894) took 2644 ms. Trace:
I20240503 02:12:44.546433 5159 rpcz_store.cc:276] 0503 02:12:41.902434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.902877 (+ 443us) service_pool.cc:225] Handling call
0503 02:12:44.546217 (+2643340us) ts_tablet_manager.cc:1120] Deleting tablet 70e08c10c15f41878202cef0c8d5ad3c
0503 02:12:44.546227 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.546328 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102,"tablet-delete.queue_time_us":2643170}
I20240503 02:12:44.546816 5159 tablet_replica.cc:330] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.547044 5159 raft_consensus.cc:2229] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.547396 5159 raft_consensus.cc:2258] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.548015 5159 ts_tablet_manager.cc:1875] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.548089 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8195767d80384a6c9a4690292f6776de (table default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]) successfully deleted
I20240503 02:12:44.545195 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7b745bb994555a3e513f92ff29429. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:44.548444 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 70e08c10c15f41878202cef0c8d5ad3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e08c10c15f41878202cef0c8d5ad3c already in progress: creating tablet
I20240503 02:12:44.548892 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cced6c58c1554d1dad38a0f553a430e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.552484 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81213d7b9fc9429abd17f93b9f6982c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.554419 5105 tablet_bootstrap.cc:654] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.555898 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61842978046d4cf5a4b123b9052bc754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.557083 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b1c210cf042838016de53574b0ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.561825 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40813cdfee114fc2ac10c45130b11f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.564325 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_8445a7ddbfe94d78a5489dd954b51d8b [id=4d9a163c0c564955a47a65f799a0018c] alter complete (version 5)
I20240503 02:12:44.564643 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 9003cc376aa149b59947bc32bd071163 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:44.565126 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 881 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:44.565569 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d578b8abd804e19a45d634721760ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.569068 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cc893b1f664fddbb2833cb0b9843bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.535615 5114 ts_tablet_manager.cc:1875] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.576447 5159 ts_tablet_manager.cc:1888] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240503 02:12:44.579960 5114 ts_tablet_manager.cc:1888] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.579977 5159 log.cc:1196] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:44.580107 5114 log.cc:1196] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/4b4972057fae45168756d7c28ddbe2bf
I20240503 02:12:44.580505 5114 ts_tablet_manager.cc:1909] T 4b4972057fae45168756d7c28ddbe2bf P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.581245 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_1a2d9fd60ab3486cbd8a1379d896f5b5" } new_table_name: "default.Table_6b228c5e68694c6a854d54eb569ffa31" modify_external_catalogs: true
I20240503 02:12:44.581816 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 93cc893b1f664fddbb2833cb0b9843bd (DEFAULT_TABLE table=default.Table_4b0e3fb2ecb24f7ea287a2cbabe12de8 [id=2e843e566e6941e5aedeeceed7c1da8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.581816 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 40813cdfee114fc2ac10c45130b11f98 (DEFAULT_TABLE table=default.Table_0f61ff50b4434dbbaae6066ce10e5f6b [id=127970bcecef4f299992bd8203ddbe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.585707 5103 tablet_bootstrap.cc:654] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.589315 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cc893b1f664fddbb2833cb0b9843bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.593217 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40813cdfee114fc2ac10c45130b11f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.593617 5159 ts_tablet_manager.cc:1909] T c88eaba979394b449d96ee956c3f28a8 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.594440 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 895) took 2690 ms. Trace:
I20240503 02:12:44.594529 5159 rpcz_store.cc:276] 0503 02:12:41.903866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.904129 (+ 263us) service_pool.cc:225] Handling call
0503 02:12:44.546766 (+2642637us) ts_tablet_manager.cc:1120] Deleting tablet c88eaba979394b449d96ee956c3f28a8
0503 02:12:44.546777 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.568146 (+ 21369us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.576330 (+ 8184us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.594417 (+ 18087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2642419}
W20240503 02:12:44.594841 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6caae07d88af48988d5619efcca508f5 consensus state has no leader
I20240503 02:12:44.595579 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]) successfully deleted
I20240503 02:12:44.597975 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_9ed74e0af4724d85a61a10acc9942836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.602434 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet d607da9689c14e51add9178a48b35913 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:44.607995 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_b48ee0fa4b364d7ca10d28cc8224840e" } new_table_name: "default.Table_c4b9771315234210a0453b55218c008e" modify_external_catalogs: true
I20240503 02:12:44.609488 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 896) took 2704 ms. Trace:
I20240503 02:12:44.612805 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 856) took 2543 ms. Trace:
W20240503 02:12:44.614674 4270 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 166599de742e42ea8733d46ead986341 consensus state has no leader
I20240503 02:12:44.408623 6698 raft_consensus.cc:482] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.616030 6698 raft_consensus.cc:504] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.616786 6698 leader_election.cc:288] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.420395 4644 tablet_service.cc:1467] Processing CreateTablet for tablet d4b24a9822cd4e3693cf7511c2dc2eb9 (DEFAULT_TABLE table=default.Table_2e9fd306667548278723f880bb82a806 [id=a37ff45de8d545f38ea07539538f7eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.617276 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b24a9822cd4e3693cf7511c2dc2eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.617419 4270 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 91798b4bc2f74b9289425531937a78ce (table default.Table_8c5ce4d70c734f70a52c3e3267097103 [id=a401eebd3ac541e19b6f3f310135a4da]). Expected version 3 got 0
I20240503 02:12:44.613085 5114 rpcz_store.cc:276] 0503 02:12:42.069554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.069599 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:44.516782 (+2447183us) ts_tablet_manager.cc:1120] Deleting tablet 4b4972057fae45168756d7c28ddbe2bf
0503 02:12:44.516793 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.576754 (+ 59961us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.579840 (+ 3086us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.612756 (+ 32916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2447035}
I20240503 02:12:44.618934 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 867) took 2373 ms. Trace:
I20240503 02:12:44.619088 5114 rpcz_store.cc:276] 0503 02:12:42.245891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.245940 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:44.618724 (+2372784us) ts_tablet_manager.cc:1120] Deleting tablet d99a7e930a3947b4aae62424f11d7227
0503 02:12:44.618735 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.618915 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372502}
I20240503 02:12:44.619835 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 866) took 2374 ms. Trace:
I20240503 02:12:44.620069 5159 rpcz_store.cc:276] 0503 02:12:41.905168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.905312 (+ 144us) service_pool.cc:225] Handling call
0503 02:12:44.594808 (+2689496us) ts_tablet_manager.cc:1120] Deleting tablet 7b86a6bb791449b5a4c490663d87afd4
0503 02:12:44.594820 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.609449 (+ 14629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689295}
I20240503 02:12:44.619958 5114 rpcz_store.cc:276] 0503 02:12:42.245788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.246369 (+ 581us) service_pool.cc:225] Handling call
0503 02:12:44.619633 (+2373264us) ts_tablet_manager.cc:1120] Deleting tablet 1a082f4c43fd4c188be9ee3966e26edf
0503 02:12:44.619670 (+ 37us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.619818 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373091}
I20240503 02:12:44.620541 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 897) took 2714 ms. Trace:
I20240503 02:12:44.620620 5159 rpcz_store.cc:276] 0503 02:12:41.906539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.906746 (+ 207us) service_pool.cc:225] Handling call
0503 02:12:44.620393 (+2713647us) ts_tablet_manager.cc:1120] Deleting tablet 7982ba6234b844f682cb23c0bfc4068c
0503 02:12:44.620406 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.620520 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2713462}
I20240503 02:12:44.420593 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 83ff682e44214c4c809ac4a12b2f23d2 (DEFAULT_TABLE table=default.Table_fc35d19d5cd54bc2824834be14a6bb14 [id=b077c9cd1617497bbbca20af7aed5f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.620980 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 898) took 2713 ms. Trace:
I20240503 02:12:44.621088 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 868) took 2374 ms. Trace:
I20240503 02:12:44.621261 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ff682e44214c4c809ac4a12b2f23d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.621235 5114 rpcz_store.cc:276] 0503 02:12:42.246432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.246477 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:44.620889 (+2374412us) ts_tablet_manager.cc:1120] Deleting tablet 42e80cc31f5c4908b82f2b030255f6af
0503 02:12:44.620898 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.621071 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2374287}
I20240503 02:12:44.621968 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 869) took 2375 ms. Trace:
I20240503 02:12:44.622072 5114 rpcz_store.cc:276] 0503 02:12:42.246798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.246838 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:44.621774 (+2374936us) ts_tablet_manager.cc:1120] Deleting tablet b7d0640cea2a482e9974376ba479163b
0503 02:12:44.621785 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.621924 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2374760}
I20240503 02:12:44.622649 5103 tablet_bootstrap.cc:492] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:44.622776 5103 ts_tablet_manager.cc:1367] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.000s
I20240503 02:12:44.622800 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 870) took 2375 ms. Trace:
I20240503 02:12:44.622924 5114 rpcz_store.cc:276] 0503 02:12:42.247148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.247188 (+ 40us) service_pool.cc:225] Handling call
0503 02:12:44.622611 (+2375423us) ts_tablet_manager.cc:1120] Deleting tablet 810d37e454cb4633a099c546925eb48c
0503 02:12:44.622620 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.622754 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375283}
I20240503 02:12:44.623519 5114 tablet_replica.cc:330] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:44.623885 5114 raft_consensus.cc:2229] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.420743 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 939e2b3230e645e881a8f926efae550e (DEFAULT_TABLE table=default.Table_2785fa72fa1e42629f30b99a8dc42925 [id=5cb67145cf554e539e98fc4ded1ebe5f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:44.624604 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ed74e0af4724d85a61a10acc9942836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.624672 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939e2b3230e645e881a8f926efae550e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.429975 6758 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 02:12:44.381310 6695 leader_election.cc:288] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.465333 5106 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.629251 5106 ts_tablet_manager.cc:1367] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.259s user 0.003s sys 0.003s
I20240503 02:12:44.630007 5106 raft_consensus.cc:348] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.630301 5106 raft_consensus.cc:374] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.630357 5106 raft_consensus.cc:729] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.630574 5106 consensus_queue.cc:260] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.631098 5106 ts_tablet_manager.cc:1398] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:44.631430 5106 tablet_bootstrap.cc:492] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:44.465605 4608 leader_election.cc:341] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.493978 6699 raft_consensus.cc:482] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.633045 6699 raft_consensus.cc:504] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.633155 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 167547us
W20240503 02:12:44.633554 4608 consensus_peers.cc:485] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.633647 6699 leader_election.cc:288] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:44.633761 4608 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.633847 4608 leader_election.cc:302] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.633857 6756 tablet.cc:1722] T 91798b4bc2f74b9289425531937a78ce P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 02:12:44.634218 4608 leader_election.cc:341] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.634479 5114 raft_consensus.cc:2258] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.635118 6758 tablet.cc:1722] T 91798b4bc2f74b9289425531937a78ce P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:44.635202 5114 ts_tablet_manager.cc:1875] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.636070 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 7219ce14a9ae4488b97914b9d02995f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.636319 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.636458 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet f146f7cce0b24d3087bfe0973e2b7ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.636603 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.637051 4652 tablet_service.cc:1467] Processing CreateTablet for tablet f14b1c210cf042838016de53574b0ef4 (DEFAULT_TABLE table=default.Table_2c4258a98cf14b69af1ffe992cf712e2 [id=e5b56ec392f74745bb748ef178755789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.637495 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14b1c210cf042838016de53574b0ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.621043 5159 rpcz_store.cc:276] 0503 02:12:41.907586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.907640 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:44.620843 (+2713203us) ts_tablet_manager.cc:1120] Deleting tablet ee16f1beb6a34a85bc872dedd64472ce
0503 02:12:44.620854 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.620964 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2713053}
I20240503 02:12:44.639544 5159 tablet_replica.cc:330] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.639794 5159 raft_consensus.cc:2229] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.639971 5159 raft_consensus.cc:2258] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.641264 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 (DEFAULT_TABLE table=default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6 [id=0f6b37021cc04323af6f820204114af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.641314 5103 raft_consensus.cc:348] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.641556 5103 raft_consensus.cc:374] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.641625 5103 raft_consensus.cc:729] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.641705 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81213d7b9fc9429abd17f93b9f6982c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.641906 5114 ts_tablet_manager.cc:1888] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:44.642035 5114 log.cc:1196] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/64cdb861bc9f448698e243718136c303
I20240503 02:12:44.642453 5114 ts_tablet_manager.cc:1909] T 64cdb861bc9f448698e243718136c303 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.645447 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 61842978046d4cf5a4b123b9052bc754 (DEFAULT_TABLE table=default.Table_15995c9366f647308e1abed005edeb79 [id=e68f9d54345643248131db5a41806ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.645907 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61842978046d4cf5a4b123b9052bc754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.647994 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 871) took 2331 ms. Trace:
I20240503 02:12:44.648113 5114 rpcz_store.cc:276] 0503 02:12:42.316977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.323849 (+ 6872us) service_pool.cc:225] Handling call
0503 02:12:44.623446 (+2299597us) ts_tablet_manager.cc:1120] Deleting tablet 64cdb861bc9f448698e243718136c303
0503 02:12:44.623455 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.638432 (+ 14977us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.641793 (+ 3361us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.647960 (+ 6167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10037,"tablet-delete.queue_time_us":2299407}
I20240503 02:12:44.648870 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 873) took 2183 ms. Trace:
I20240503 02:12:44.649003 5114 rpcz_store.cc:276] 0503 02:12:42.465291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.465347 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:44.648721 (+2183374us) ts_tablet_manager.cc:1120] Deleting tablet 5a0303f1b1194115a7142a00470457c3
0503 02:12:44.648734 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.648851 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183170}
I20240503 02:12:44.649760 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 6d578b8abd804e19a45d634721760ff7 (DEFAULT_TABLE table=default.Table_9df571d7b21a4cce8f3f5423c9d50a37 [id=38262bbd9346419e853465e448fbafe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.649864 5114 tablet_replica.cc:330] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:44.650069 5114 raft_consensus.cc:2229] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.650447 5114 raft_consensus.cc:2258] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.641837 5103 consensus_queue.cc:260] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.651190 5103 ts_tablet_manager.cc:1398] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:12:44.651464 5114 ts_tablet_manager.cc:1875] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.652295 4637 tablet_service.cc:1467] Processing CreateTablet for tablet cced6c58c1554d1dad38a0f553a430e1 (DEFAULT_TABLE table=default.Table_84073519a2f04b50863dc1ea8bac73b9 [id=dc351d99b5e54398adecc427a44cbfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.652381 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 96a655b2dbdd4b9091584c6cf8a8d907 (DEFAULT_TABLE table=default.Table_4a279b299cf046659678db65a1ee5ddc [id=c60587696f8f4d3cb1d74085fd8de267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.653517 4646 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7b745bb994555a3e513f92ff29429 (DEFAULT_TABLE table=default.Table_ffa48032226a4aa4859ee9f4a330014b [id=4acf29d3dbe1430398cb852644c51a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.654776 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c13f257eae44689af07cc31ea254c4" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.655607 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 40813cdfee114fc2ac10c45130b11f98 (DEFAULT_TABLE table=default.Table_0f61ff50b4434dbbaae6066ce10e5f6b [id=127970bcecef4f299992bd8203ddbe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.656018 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 93cc893b1f664fddbb2833cb0b9843bd (DEFAULT_TABLE table=default.Table_4b0e3fb2ecb24f7ea287a2cbabe12de8 [id=2e843e566e6941e5aedeeceed7c1da8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.656337 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 9003cc376aa149b59947bc32bd071163 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.656533 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet f40706400f114def9e14e6d1d7114962 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.634223 6699 raft_consensus.cc:2726] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.659312 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f88d274b7654ba4942e757ba3302101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.660122 5103 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:44.660496 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4940fd3c16424eaaaef59ab50a04a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:44.661700 5103 tablet_bootstrap.cc:654] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.666733 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:44.666986 4810 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:44.668515 5159 ts_tablet_manager.cc:1875] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.681149 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.681946 5106 tablet_bootstrap.cc:654] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.696568 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8e5698a54a9249b2a78432ae0801f067" } modify_external_catalogs: true
I20240503 02:12:44.696847 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8e5698a54a9249b2a78432ae0801f067" } modify_external_catalogs: true
I20240503 02:12:44.698622 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_1ad83b282d514dc290b287c145589308"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.699306 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ad83b282d514dc290b287c145589308 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.700819 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_adfe1adf426343a2ace07860d42d9083"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.701434 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adfe1adf426343a2ace07860d42d9083 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.694731 4518 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:44.706745 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_74b07d82f6514260885515c05151bec9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.707453 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74b07d82f6514260885515c05151bec9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.708302 6619 raft_consensus.cc:482] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.708285 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d0a57800fa3a48808a5f01bd6bb4945b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.708472 6619 raft_consensus.cc:504] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.708854 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0a57800fa3a48808a5f01bd6bb4945b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.709354 6619 leader_election.cc:288] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.710259 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.710496 4667 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:44.710716 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5f1814596fb2492f9b62c8d2befb76e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:44.710783 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2ff3d2b4b7054749b3d29eefd67d6abf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.711298 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f1814596fb2492f9b62c8d2befb76e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.711259 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:44.711321 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ff3d2b4b7054749b3d29eefd67d6abf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.711459 4805 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:44.712355 4462 leader_election.cc:302] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.712817 6619 raft_consensus.cc:2781] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.712883 6619 raft_consensus.cc:482] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.712965 6619 raft_consensus.cc:3037] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.715113 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2f9fb44984074b81baaac14bf2486c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:44.715731 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f9fb44984074b81baaac14bf2486c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.715649 6619 raft_consensus.cc:504] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.716300 6619 leader_election.cc:288] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.716877 4275 master_service.cc:946] ReplaceTablet: received request to replace tablet 79aae500e9a842ed90357f4a65fd37ba from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:44.719527 5114 ts_tablet_manager.cc:1888] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.719731 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:44.719933 4667 raft_consensus.cc:3037] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.722699 4667 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:44.723780 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dfb4150c3c49f4b5d4e31396da4e8d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:44.724017 4807 raft_consensus.cc:3037] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.724443 5114 log.cc:1196] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/8d0494ee4bd746069bfaf39bfb46687d
I20240503 02:12:44.723850 4462 leader_election.cc:302] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.726941 4807 raft_consensus.cc:2454] T d9dfb4150c3c49f4b5d4e31396da4e8d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:44.727149 5159 ts_tablet_manager.cc:1888] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.729405 5114 ts_tablet_manager.cc:1909] T 8d0494ee4bd746069bfaf39bfb46687d P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.734391 6619 raft_consensus.cc:2781] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.739048 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 874) took 2272 ms. Trace:
I20240503 02:12:44.741355 6619 raft_consensus.cc:686] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:44.741307 5114 rpcz_store.cc:276] 0503 02:12:42.466191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.466240 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:44.649794 (+2183554us) ts_tablet_manager.cc:1120] Deleting tablet 8d0494ee4bd746069bfaf39bfb46687d
0503 02:12:44.649814 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.655159 (+ 5345us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.719376 (+ 64217us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.739014 (+ 19638us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":2183379}
I20240503 02:12:44.741616 6619 consensus_queue.cc:237] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.741990 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 875) took 2271 ms. Trace:
I20240503 02:12:44.742080 5114 rpcz_store.cc:276] 0503 02:12:42.470416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.470467 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:44.741844 (+2271377us) ts_tablet_manager.cc:1120] Deleting tablet 01c61bf67c8a47fb9ef23bcd6c58c448
0503 02:12:44.741857 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.741971 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270864}
I20240503 02:12:44.742372 5114 tablet_replica.cc:330] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:44.742568 5114 raft_consensus.cc:2229] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:44.742866 5114 raft_consensus.cc:2258] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.745023 5159 log.cc:1196] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/88b173d78ba74ede801b12cccbeef6b1
I20240503 02:12:44.745489 5159 ts_tablet_manager.cc:1909] T 88b173d78ba74ede801b12cccbeef6b1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:44.746251 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 904) took 2690 ms. Trace:
W20240503 02:12:44.754320 4608 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:44.756763 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7b86a6bb791449b5a4c490663d87afd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b86a6bb791449b5a4c490663d87afd4 already in progress: creating tablet
I20240503 02:12:44.765368 5114 ts_tablet_manager.cc:1875] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.765702 5159 rpcz_store.cc:276] 0503 02:12:42.056008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.064980 (+ 8972us) service_pool.cc:225] Handling call
0503 02:12:44.639475 (+2574495us) ts_tablet_manager.cc:1120] Deleting tablet 88b173d78ba74ede801b12cccbeef6b1
0503 02:12:44.639489 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.671867 (+ 32378us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.727029 (+ 55162us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.746232 (+ 19203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574304}
I20240503 02:12:44.768023 4608 leader_election.cc:302] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.769151 5159 tablet_replica.cc:330] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:44.769881 5159 raft_consensus.cc:2229] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:44.771605 5159 raft_consensus.cc:2258] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:44.772056 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7982ba6234b844f682cb23c0bfc4068c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7982ba6234b844f682cb23c0bfc4068c already in progress: creating tablet
W20240503 02:12:44.772413 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ee16f1beb6a34a85bc872dedd64472ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee16f1beb6a34a85bc872dedd64472ce already in progress: creating tablet
I20240503 02:12:44.772620 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 88b173d78ba74ede801b12cccbeef6b1 (table default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]) successfully deleted
I20240503 02:12:44.772837 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 4b4972057fae45168756d7c28ddbe2bf (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]) successfully deleted
W20240503 02:12:44.773077 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet d99a7e930a3947b4aae62424f11d7227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d99a7e930a3947b4aae62424f11d7227 already in progress: creating tablet
W20240503 02:12:44.773639 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1a082f4c43fd4c188be9ee3966e26edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a082f4c43fd4c188be9ee3966e26edf already in progress: creating tablet
W20240503 02:12:44.774335 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 42e80cc31f5c4908b82f2b030255f6af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42e80cc31f5c4908b82f2b030255f6af already in progress: creating tablet
W20240503 02:12:44.774956 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b7d0640cea2a482e9974376ba479163b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d0640cea2a482e9974376ba479163b already in progress: creating tablet
W20240503 02:12:44.775574 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 810d37e454cb4633a099c546925eb48c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810d37e454cb4633a099c546925eb48c already in progress: creating tablet
I20240503 02:12:44.777181 6762 raft_consensus.cc:2741] T d9dfb4150c3c49f4b5d4e31396da4e8d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20240503 02:12:44.777271 4608 consensus_peers.cc:485] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9511e4de6c942b9b9e0790c88d3d543. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.698575 5103 tablet_bootstrap.cc:492] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:44.779330 5103 ts_tablet_manager.cc:1367] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.119s user 0.006s sys 0.000s
I20240503 02:12:44.781385 5103 raft_consensus.cc:348] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.803234 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d578b8abd804e19a45d634721760ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.803550 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7b745bb994555a3e513f92ff29429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.803761 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a655b2dbdd4b9091584c6cf8a8d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.806957 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40813cdfee114fc2ac10c45130b11f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.809094 5159 ts_tablet_manager.cc:1875] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:44.813062 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cc893b1f664fddbb2833cb0b9843bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:44.819388 4608 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 27c13f257eae44689af07cc31ea254c4
I20240503 02:12:44.819535 4608 leader_election.cc:302] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:44.819784 6762 raft_consensus.cc:2726] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.821028 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cced6c58c1554d1dad38a0f553a430e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.821347 5106 tablet_bootstrap.cc:492] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.821471 5106 ts_tablet_manager.cc:1367] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.190s user 0.004s sys 0.000s
I20240503 02:12:44.781991 5103 raft_consensus.cc:374] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.822048 5103 raft_consensus.cc:729] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.822353 5103 consensus_queue.cc:260] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.822896 5103 ts_tablet_manager.cc:1398] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20240503 02:12:44.823199 5103 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:44.776715 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]) successfully deleted
W20240503 02:12:44.829406 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 52780.9us
I20240503 02:12:44.830452 5106 raft_consensus.cc:348] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.830654 5106 raft_consensus.cc:374] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.830699 5106 raft_consensus.cc:729] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.830862 5106 consensus_queue.cc:260] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.831545 5106 ts_tablet_manager.cc:1398] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:12:44.831876 5106 tablet_bootstrap.cc:492] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.833722 5106 tablet_bootstrap.cc:654] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.834169 5115 raft_consensus.cc:2258] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:44.834853 5115 ts_tablet_manager.cc:1875] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:44.851462 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 5a0303f1b1194115a7142a00470457c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a0303f1b1194115a7142a00470457c3 already in progress: creating tablet
I20240503 02:12:44.851856 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 8d0494ee4bd746069bfaf39bfb46687d (table default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]) successfully deleted
I20240503 02:12:44.852337 5114 ts_tablet_manager.cc:1888] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.858681 5114 log.cc:1196] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/43119471f91c4a1ba900f267c88db9a6
W20240503 02:12:44.849001 4608 consensus_peers.cc:485] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8195767d80384a6c9a4690292f6776de. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.860374 5114 ts_tablet_manager.cc:1909] T 43119471f91c4a1ba900f267c88db9a6 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:44.860687 6738 raft_consensus.cc:482] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.860934 6762 consensus_queue.cc:1035] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.860898 6738 raft_consensus.cc:504] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:44.861775 4608 consensus_peers.cc:485] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b8db679ba24ac6a93be591c0513ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.861830 6738 leader_election.cc:288] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.863194 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94559cfc32704eae96b121f165686edc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:44.863843 4746 leader_election.cc:341] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:44.869537 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c61bf67c8a47fb9ef23bcd6c58c448 already in progress: creating tablet
I20240503 02:12:44.869830 5105 tablet_bootstrap.cc:492] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:44.869967 5105 ts_tablet_manager.cc:1367] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.325s user 0.006s sys 0.028s
I20240503 02:12:44.870746 5105 raft_consensus.cc:348] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.870940 5105 raft_consensus.cc:374] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.871001 5105 raft_consensus.cc:729] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.871205 5105 consensus_queue.cc:260] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.871809 5105 ts_tablet_manager.cc:1398] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:44.872124 5105 tablet_bootstrap.cc:492] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:44.877084 6757 tablet.cc:1722] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 02:12:44.879127 5105 tablet_bootstrap.cc:654] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.884384 5159 ts_tablet_manager.cc:1888] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:44.864100 6619 raft_consensus.cc:482] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.885507 6619 raft_consensus.cc:504] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.887462 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035216c9475d4b59aa1ac81b285b4d8a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.887799 4667 raft_consensus.cc:2454] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:44.888585 4462 leader_election.cc:302] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.889030 6769 raft_consensus.cc:2781] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.889156 6769 raft_consensus.cc:482] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.889240 6769 raft_consensus.cc:3037] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.893771 6769 raft_consensus.cc:504] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.895074 6769 leader_election.cc:288] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.895977 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035216c9475d4b59aa1ac81b285b4d8a" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:44.896196 4667 raft_consensus.cc:3037] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.901082 4667 raft_consensus.cc:2454] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:44.902197 4462 leader_election.cc:302] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:44.902668 4462 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:44.437909 6226 ts_manager.cc:194] Registered new tserver with Master: acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.447937 6228 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1f50719d94564844b15e1a9211d48a81" instance_seqno: 1714702334548221) as {username='slave'} at 127.4.17.66:56341; Asking this server to re-register.
I20240503 02:12:44.504225 6227 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" instance_seqno: 1714702333742995) as {username='slave'} at 127.4.17.65:46199; Asking this server to re-register.
I20240503 02:12:44.573762 6193 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:44.912122 4574 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:44.915357 4720 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:44.917272 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94559cfc32704eae96b121f165686edc" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:44.919111 5106 tablet_bootstrap.cc:492] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.871840 5103 tablet_bootstrap.cc:654] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.879552 6620 raft_consensus.cc:482] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.919850 5106 ts_tablet_manager.cc:1367] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.088s user 0.022s sys 0.023s
I20240503 02:12:44.919847 6620 raft_consensus.cc:504] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.919958 6756 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 02:12:44.920544 6620 leader_election.cc:288] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.920684 6620 raft_consensus.cc:2781] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.920773 6620 raft_consensus.cc:686] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:44.922744 5106 raft_consensus.cc:348] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.922997 5106 raft_consensus.cc:374] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.923070 5106 raft_consensus.cc:729] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.923305 5106 consensus_queue.cc:260] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.924036 5106 ts_tablet_manager.cc:1398] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:44.924599 5106 tablet_bootstrap.cc:492] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.925287 6620 consensus_queue.cc:237] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.927346 5106 tablet_bootstrap.cc:654] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.929078 6226 ts_manager.cc:194] Registered new tserver with Master: f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:44.937309 5106 tablet_bootstrap.cc:492] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.938102 5106 ts_tablet_manager.cc:1367] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20240503 02:12:44.938823 6227 ts_manager.cc:194] Registered new tserver with Master: 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:44.939779 5106 raft_consensus.cc:348] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.940876 5106 raft_consensus.cc:374] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.941144 5106 raft_consensus.cc:729] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.941516 5106 consensus_queue.cc:260] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.942740 5106 ts_tablet_manager.cc:1398] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:44.943631 5106 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.943732 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b149e66a0446eeade190dd7f5d2bec" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:44.944336 4667 raft_consensus.cc:2454] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
W20240503 02:12:44.945546 4608 consensus_peers.cc:485] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:44.946652 4462 leader_election.cc:302] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:44.947707 5106 tablet_bootstrap.cc:654] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.953243 6762 consensus_queue.cc:1035] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.956399 6769 raft_consensus.cc:2781] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.956542 6769 raft_consensus.cc:482] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.956658 6769 raft_consensus.cc:3037] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.957533 5106 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:44.958092 5106 ts_tablet_manager.cc:1367] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
W20240503 02:12:44.958748 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 166599de742e42ea8733d46ead986341 (table default.Table_c4b9771315234210a0453b55218c008e [id=3cce450e263f4c38bff14b82f72f8470], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 166599de742e42ea8733d46ead986341 consensus state has no leader
I20240503 02:12:44.960516 5106 raft_consensus.cc:348] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.960794 6699 consensus_queue.cc:1035] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.961422 5106 raft_consensus.cc:374] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.961618 5106 raft_consensus.cc:729] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.961933 5106 consensus_queue.cc:260] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.964260 6620 consensus_queue.cc:1035] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.975241 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_cf90801bade4413b98880d6506b87076" } new_table_name: "default.Table_b5f830205c614d6e8b6ad4deaa6f01a4" modify_external_catalogs: true
I20240503 02:12:44.977391 5103 tablet_bootstrap.cc:492] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:44.977721 5103 ts_tablet_manager.cc:1367] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.001s
I20240503 02:12:44.978533 5103 raft_consensus.cc:348] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.978808 5103 raft_consensus.cc:374] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.978868 5103 raft_consensus.cc:729] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.979066 5103 consensus_queue.cc:260] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.981102 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 (DEFAULT_TABLE table=default.Table_eee9e761694041d69f22b08ae4363113 [id=a551765407ab434fa2f1e8161e9df406]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.981637 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.984442 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 1885a24e55854760a30ef7249f1dc8c6 (DEFAULT_TABLE table=default.Table_7dbc61340b924e26bc747d93095af77b [id=629de0c5eba546c08e2113ad36871874]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.984570 5106 ts_tablet_manager.cc:1398] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:12:44.984841 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885a24e55854760a30ef7249f1dc8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.984871 5106 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.986506 5106 tablet_bootstrap.cc:654] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.991494 6619 leader_election.cc:288] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:44.994542 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 (DEFAULT_TABLE table=default.Table_eee9e761694041d69f22b08ae4363113 [id=a551765407ab434fa2f1e8161e9df406]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:44.918773 4746 leader_election.cc:341] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:44.997201 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 8d136c1e5b27492385f428bdf988e588 (DEFAULT_TABLE table=default.Table_d3383012c69d498887d4cb8f518bcdd6 [id=9fc23e541c444bcfa620a68051d0b0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.997695 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d136c1e5b27492385f428bdf988e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.976030 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_ca7d5d73ff93496b90bdb5aebca4cde5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:45.000074 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 3478033c4ed94268a2436a2942c27a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:45.000176 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca7d5d73ff93496b90bdb5aebca4cde5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:44.996321 4746 leader_election.cc:302] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:45.000775 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 82020.4us
I20240503 02:12:44.976753 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_b7257d81b4094fc495ca99d263034757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.001647 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7257d81b4094fc495ca99d263034757 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.001689 6739 raft_consensus.cc:2726] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:44.980324 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 (DEFAULT_TABLE table=default.Table_eee9e761694041d69f22b08ae4363113 [id=a551765407ab434fa2f1e8161e9df406]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.980641 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 1885a24e55854760a30ef7249f1dc8c6 (DEFAULT_TABLE table=default.Table_7dbc61340b924e26bc747d93095af77b [id=629de0c5eba546c08e2113ad36871874]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.002116 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.002359 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885a24e55854760a30ef7249f1dc8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.977383 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_6616d554610d429190adcee205cf086c" } new_table_name: "default.Table_375af786345e47b69fba154241e880c7" modify_external_catalogs: true
I20240503 02:12:44.995818 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 8d136c1e5b27492385f428bdf988e588 (DEFAULT_TABLE table=default.Table_d3383012c69d498887d4cb8f518bcdd6 [id=9fc23e541c444bcfa620a68051d0b0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.977880 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_4d9a0ac169f94dfbad1496820ded5f07" } new_table_name: "default.Table_637aa20655eb485f921eeebcadeee3eb" modify_external_catalogs: true
I20240503 02:12:45.003867 5106 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.003999 5106 ts_tablet_manager.cc:1367] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20240503 02:12:45.004072 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d136c1e5b27492385f428bdf988e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.980157 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 79aae500e9a842ed90357f4a65fd37ba
I20240503 02:12:45.004760 5106 raft_consensus.cc:348] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.004987 5106 raft_consensus.cc:374] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.005076 5106 raft_consensus.cc:729] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.005319 5106 consensus_queue.cc:260] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.005477 4275 catalog_manager.cc:6554] ReplaceTablet: tablet 79aae500e9a842ed90357f4a65fd37ba deleted and replaced by tablet a673151d0db649a0914141f7623d11ed
I20240503 02:12:45.005810 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b" } modify_external_catalogs: true
I20240503 02:12:45.006001 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_55aa73f8c53a49b5a8c8e68a7beb7a0b" } modify_external_catalogs: true
I20240503 02:12:45.006057 5106 ts_tablet_manager.cc:1398] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:45.006350 5106 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:44.981993 4271 catalog_manager.cc:5484] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:45.007930 5106 tablet_bootstrap.cc:654] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.007957 5159 log.cc:1196] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8d0494ee4bd746069bfaf39bfb46687d
I20240503 02:12:45.008189 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:45.008358 5159 ts_tablet_manager.cc:1909] T 8d0494ee4bd746069bfaf39bfb46687d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:45.009279 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 905) took 2953 ms. Trace:
I20240503 02:12:45.015239 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 883) took 2397 ms. Trace:
I20240503 02:12:45.015870 4267 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a9135679f7fb423cb1d8d69c3430feb0" } modify_external_catalogs: true
I20240503 02:12:45.016072 4267 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a9135679f7fb423cb1d8d69c3430feb0" } modify_external_catalogs: true
I20240503 02:12:45.017470 6762 raft_consensus.cc:482] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:45.017628 6762 raft_consensus.cc:504] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:45.018221 6762 leader_election.cc:288] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.019400 4643 tablet_service.cc:1467] Processing CreateTablet for tablet 1885a24e55854760a30ef7249f1dc8c6 (DEFAULT_TABLE table=default.Table_7dbc61340b924e26bc747d93095af77b [id=629de0c5eba546c08e2113ad36871874]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.019976 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 8d136c1e5b27492385f428bdf988e588 (DEFAULT_TABLE table=default.Table_d3383012c69d498887d4cb8f518bcdd6 [id=9fc23e541c444bcfa620a68051d0b0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.024891 6762 consensus_queue.cc:1035] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:45.044010 4608 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 95681e67db1e4485876d530133e8a514
I20240503 02:12:45.048379 6762 raft_consensus.cc:482] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.048542 6762 raft_consensus.cc:504] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.049191 6762 leader_election.cc:288] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.052995 5114 rpcz_store.cc:276] 0503 02:12:42.618165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.627161 (+ 8996us) service_pool.cc:225] Handling call
0503 02:12:44.742315 (+2115154us) ts_tablet_manager.cc:1120] Deleting tablet 43119471f91c4a1ba900f267c88db9a6
0503 02:12:44.742326 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.807438 (+ 65112us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.852188 (+ 44750us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.014485 (+162297us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x61800000fd18. stack: 00007f139331d756 00007f139331d510 00007f139235048d 0000000000955804 00000000009557cd 000000000095519a 00007f139331319d 00007f139331420e 00007f13a681a4ed 00007f13a67966b3 00007f13a28f99dd 00007f13a292b8d4 00007f13a290cbc0 00007f13a290c94d 0000000000a60bcc 00007f1393363b74
0503 02:12:45.015203 (+ 718us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":296954624,"tablet-delete.queue_time_us":2114395}
I20240503 02:12:45.055199 6762 raft_consensus.cc:482] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.055382 6762 raft_consensus.cc:504] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.021656 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aae500e9a842ed90357f4a65fd37ba with delete_type TABLET_DATA_DELETED (replaced by tablet a673151d0db649a0914141f7623d11ed) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.037789 4497 tablet_service.cc:1467] Processing CreateTablet for tablet f8b20a0190ce4ed2aacd1f9e3fc26446 (DEFAULT_TABLE table=default.Table_cf194cb302824ebe8a359b8225c7725a [id=ef5525e121cf47a985aa41e5b997c63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.056253 6762 leader_election.cc:288] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.056603 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b20a0190ce4ed2aacd1f9e3fc26446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.057761 4637 tablet_service.cc:1467] Processing CreateTablet for tablet f8b20a0190ce4ed2aacd1f9e3fc26446 (DEFAULT_TABLE table=default.Table_cf194cb302824ebe8a359b8225c7725a [id=ef5525e121cf47a985aa41e5b997c63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.057761 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 6d74dcf665da4dfd9116638b298913b5 (DEFAULT_TABLE table=default.Table_aca5c69ca17445739ee6c15a01908c4b [id=ff9ddac47be948f0af7386f3d19d5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.058462 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aae500e9a842ed90357f4a65fd37ba with delete_type TABLET_DATA_DELETED (replaced by tablet a673151d0db649a0914141f7623d11ed) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.038178 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 6d74dcf665da4dfd9116638b298913b5 (DEFAULT_TABLE table=default.Table_aca5c69ca17445739ee6c15a01908c4b [id=ff9ddac47be948f0af7386f3d19d5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.059538 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d74dcf665da4dfd9116638b298913b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.041103 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe121a1b274d91a0a91d45d6642144 (DEFAULT_TABLE table=default.Table_cd9386a1ac724110944b2963851c7442 [id=ce4674ca63094d70bf4865814e43c232]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.063347 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d136c1e5b27492385f428bdf988e588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.063428 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe121a1b274d91a0a91d45d6642144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.063596 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885a24e55854760a30ef7249f1dc8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.063997 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b20a0190ce4ed2aacd1f9e3fc26446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.064581 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.041560 4499 tablet_service.cc:1467] Processing CreateTablet for tablet d7c75576a7754c8faba12812f5588f93 (DEFAULT_TABLE table=default.Table_3da2b86b13c8443c8c6156273fb9d097 [id=6606e9fc8f6743d29b4eb90eafb2de59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.069182 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d74dcf665da4dfd9116638b298913b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.069505 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c75576a7754c8faba12812f5588f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.042030 4500 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb6432296c4a45850a96c3651ab7bd (DEFAULT_TABLE table=default.Table_4511fe14eef348aa8f91987405ac7bfb [id=562a3fed23794d9d8ca6ae700b979883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.042515 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 3873d0d3a0da4c7196d80148e86301f1 (DEFAULT_TABLE table=default.Table_74ec097848244b30973ad02961628793 [id=f43982ac942d47658f6124db25065bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.073388 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3873d0d3a0da4c7196d80148e86301f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.043334 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc0f45293fd48dfb1378a8c98b97dcf (DEFAULT_TABLE table=default.Table_0dc297ee46134fc6ba917b6c48a7a612 [id=c5576d0d4c1f44be929f818c81ce368c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.077412 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc0f45293fd48dfb1378a8c98b97dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.043740 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 543c396fb7584cf3bcdcadf73cc8f5e5 (DEFAULT_TABLE table=default.Table_103a11fe81534cdbb336f9b1b18bfb33 [id=6760740172574bc6b9322db6e9d52184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.081277 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543c396fb7584cf3bcdcadf73cc8f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.044245 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 38c24755244c47ba982f753b3f609119 (DEFAULT_TABLE table=default.Table_cf51dfd849f741bcb578d641aee31de4 [id=bc22ad4319f74d3b96f932858abed658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.084489 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c24755244c47ba982f753b3f609119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.047458 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 827c062351e343b8b2fcc3811d5f305b (DEFAULT_TABLE table=default.Table_6ce4d17d45734d5483fa95f40b455dd6 [id=913fd4417ab04b3a9100e2c8e574e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.090451 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827c062351e343b8b2fcc3811d5f305b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.047976 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 7833167a80c34f3e85bf9e79185ad84a (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.093389 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7833167a80c34f3e85bf9e79185ad84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.097390 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb6432296c4a45850a96c3651ab7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.111443 5115 ts_tablet_manager.cc:1888] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:45.111658 5115 log.cc:1196] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/a74f4816cad04a9eb5934aaba86c2fbc
I20240503 02:12:45.112388 5115 ts_tablet_manager.cc:1909] T a74f4816cad04a9eb5934aaba86c2fbc P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:45.112823 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 761) took 5244 ms. Trace:
I20240503 02:12:45.112963 5115 rpcz_store.cc:276] 0503 02:12:39.867899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.868328 (+ 429us) service_pool.cc:225] Handling call
0503 02:12:44.408115 (+4539787us) ts_tablet_manager.cc:1120] Deleting tablet a74f4816cad04a9eb5934aaba86c2fbc
0503 02:12:44.408124 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.973203 (+565079us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.111301 (+138098us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.112793 (+ 1492us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7260,"tablet-delete.queue_time_us":4539666}
I20240503 02:12:45.113740 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 762) took 5234 ms. Trace:
I20240503 02:12:45.113891 5115 rpcz_store.cc:276] 0503 02:12:39.878864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.878912 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:45.113526 (+5234614us) ts_tablet_manager.cc:1120] Deleting tablet 44707b6439b141b7b63f49f34f25b37a
0503 02:12:45.113545 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.113708 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5234331}
I20240503 02:12:45.114387 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 763) took 5230 ms. Trace:
I20240503 02:12:45.114500 5115 rpcz_store.cc:276] 0503 02:12:39.884280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.884337 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:45.114219 (+5229882us) ts_tablet_manager.cc:1120] Deleting tablet ab093285c39b4ce7a0bb3de9d016a918
0503 02:12:45.114230 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.114331 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5229686}
I20240503 02:12:45.114881 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 764) took 5220 ms. Trace:
I20240503 02:12:45.114984 5115 rpcz_store.cc:276] 0503 02:12:39.894377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.894443 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:45.114765 (+5220322us) ts_tablet_manager.cc:1120] Deleting tablet 4fb59bbf7b68431c86d88f27ef2528b0
0503 02:12:45.114773 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.114863 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5220150}
I20240503 02:12:45.115559 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 766) took 5158 ms. Trace:
I20240503 02:12:45.115701 5115 rpcz_store.cc:276] 0503 02:12:39.957159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.957225 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:45.115267 (+5158042us) ts_tablet_manager.cc:1120] Deleting tablet 95c7e4a2954e4161b9acc029873f923b
0503 02:12:45.115278 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.115526 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5157840}
I20240503 02:12:45.116119 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 765) took 5159 ms. Trace:
I20240503 02:12:45.116219 5115 rpcz_store.cc:276] 0503 02:12:39.957064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:39.957588 (+ 524us) service_pool.cc:225] Handling call
0503 02:12:45.115994 (+5158406us) ts_tablet_manager.cc:1120] Deleting tablet 41531ddcbf66465094d8f06e1116b3be
0503 02:12:45.116003 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.116102 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5158260}
I20240503 02:12:45.116552 5115 tablet_replica.cc:330] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:45.116817 5115 raft_consensus.cc:2229] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:45.118142 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc0f45293fd48dfb1378a8c98b97dcf (DEFAULT_TABLE table=default.Table_0dc297ee46134fc6ba917b6c48a7a612 [id=c5576d0d4c1f44be929f818c81ce368c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.118736 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc0f45293fd48dfb1378a8c98b97dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.128582 4643 tablet_service.cc:1467] Processing CreateTablet for tablet d7c75576a7754c8faba12812f5588f93 (DEFAULT_TABLE table=default.Table_3da2b86b13c8443c8c6156273fb9d097 [id=6606e9fc8f6743d29b4eb90eafb2de59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.129341 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c75576a7754c8faba12812f5588f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.133538 4636 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb6432296c4a45850a96c3651ab7bd (DEFAULT_TABLE table=default.Table_4511fe14eef348aa8f91987405ac7bfb [id=562a3fed23794d9d8ca6ae700b979883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.134220 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb6432296c4a45850a96c3651ab7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.141391 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 3873d0d3a0da4c7196d80148e86301f1 (DEFAULT_TABLE table=default.Table_74ec097848244b30973ad02961628793 [id=f43982ac942d47658f6124db25065bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.142450 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3873d0d3a0da4c7196d80148e86301f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.049702 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 694b59d610264a6183a99cf0c76ba66e (DEFAULT_TABLE table=default.Table_ccb80b3d1e0a41e282ba3a1039347590 [id=b1e60750569846dcb9b4cddb6f7f16f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.053225 5103 ts_tablet_manager.cc:1398] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20240503 02:12:45.143992 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694b59d610264a6183a99cf0c76ba66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.144358 5103 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.146427 5103 tablet_bootstrap.cc:654] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.021797 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aae500e9a842ed90357f4a65fd37ba with delete_type TABLET_DATA_DELETED (replaced by tablet a673151d0db649a0914141f7623d11ed) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.036753 4791 tablet_service.cc:1467] Processing CreateTablet for tablet f8b20a0190ce4ed2aacd1f9e3fc26446 (DEFAULT_TABLE table=default.Table_cf194cb302824ebe8a359b8225c7725a [id=ef5525e121cf47a985aa41e5b997c63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.158812 5159 rpcz_store.cc:276] 0503 02:12:42.056082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.065212 (+ 9130us) service_pool.cc:225] Handling call
0503 02:12:44.769078 (+2703866us) ts_tablet_manager.cc:1120] Deleting tablet 8d0494ee4bd746069bfaf39bfb46687d
0503 02:12:44.769095 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:44.812258 (+ 43163us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:44.884245 (+ 71987us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.009249 (+125004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703744}
I20240503 02:12:45.159368 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b20a0190ce4ed2aacd1f9e3fc26446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.159432 5159 tablet_replica.cc:330] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:45.159713 5159 raft_consensus.cc:2229] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.159870 5159 raft_consensus.cc:2258] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.160566 5159 ts_tablet_manager.cc:1875] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.162580 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 884) took 2458 ms. Trace:
I20240503 02:12:45.162782 5114 rpcz_store.cc:276] 0503 02:12:42.704035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.704090 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:45.162398 (+2458308us) ts_tablet_manager.cc:1120] Deleting tablet 9e4dabd6e05948b98f02515023e2aaa1
0503 02:12:45.162412 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.162554 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458101}
I20240503 02:12:45.163421 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 891) took 2159 ms. Trace:
I20240503 02:12:45.163581 5114 rpcz_store.cc:276] 0503 02:12:43.003856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.003938 (+ 82us) service_pool.cc:225] Handling call
0503 02:12:45.163300 (+2159362us) ts_tablet_manager.cc:1120] Deleting tablet b5f444b22112407ca02fe86ac50167a9
0503 02:12:45.163311 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.163405 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159005}
I20240503 02:12:45.164083 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 892) took 2159 ms. Trace:
I20240503 02:12:45.164228 5114 rpcz_store.cc:276] 0503 02:12:43.004342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.013520 (+ 9178us) service_pool.cc:225] Handling call
0503 02:12:45.163983 (+2150463us) ts_tablet_manager.cc:1120] Deleting tablet a6de1b96c15c4b44809a03ef99e077ba
0503 02:12:45.163993 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.164069 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150280}
I20240503 02:12:45.164685 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 899) took 2075 ms. Trace:
I20240503 02:12:45.037654 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 6d74dcf665da4dfd9116638b298913b5 (DEFAULT_TABLE table=default.Table_aca5c69ca17445739ee6c15a01908c4b [id=ff9ddac47be948f0af7386f3d19d5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.038517 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe121a1b274d91a0a91d45d6642144 (DEFAULT_TABLE table=default.Table_cd9386a1ac724110944b2963851c7442 [id=ce4674ca63094d70bf4865814e43c232]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.165422 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe121a1b274d91a0a91d45d6642144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.038820 4775 tablet_service.cc:1467] Processing CreateTablet for tablet d7c75576a7754c8faba12812f5588f93 (DEFAULT_TABLE table=default.Table_3da2b86b13c8443c8c6156273fb9d097 [id=6606e9fc8f6743d29b4eb90eafb2de59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.169427 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c75576a7754c8faba12812f5588f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.171788 6805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240503 02:12:45.172353 4462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 216198us
I20240503 02:12:45.039232 4784 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb6432296c4a45850a96c3651ab7bd (DEFAULT_TABLE table=default.Table_4511fe14eef348aa8f91987405ac7bfb [id=562a3fed23794d9d8ca6ae700b979883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.039532 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 3873d0d3a0da4c7196d80148e86301f1 (DEFAULT_TABLE table=default.Table_74ec097848244b30973ad02961628793 [id=f43982ac942d47658f6124db25065bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.040073 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc0f45293fd48dfb1378a8c98b97dcf (DEFAULT_TABLE table=default.Table_0dc297ee46134fc6ba917b6c48a7a612 [id=c5576d0d4c1f44be929f818c81ce368c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.040305 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 543c396fb7584cf3bcdcadf73cc8f5e5 (DEFAULT_TABLE table=default.Table_103a11fe81534cdbb336f9b1b18bfb33 [id=6760740172574bc6b9322db6e9d52184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.040683 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 694b59d610264a6183a99cf0c76ba66e (DEFAULT_TABLE table=default.Table_ccb80b3d1e0a41e282ba3a1039347590 [id=b1e60750569846dcb9b4cddb6f7f16f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.173290 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb6432296c4a45850a96c3651ab7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.173552 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694b59d610264a6183a99cf0c76ba66e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.173489 4462 leader_election.cc:341] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 53b149e66a0446eeade190dd7f5d2bec
I20240503 02:12:45.041435 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 38c24755244c47ba982f753b3f609119 (DEFAULT_TABLE table=default.Table_cf51dfd849f741bcb578d641aee31de4 [id=bc22ad4319f74d3b96f932858abed658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.042971 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 827c062351e343b8b2fcc3811d5f305b (DEFAULT_TABLE table=default.Table_6ce4d17d45734d5483fa95f40b455dd6 [id=913fd4417ab04b3a9100e2c8e574e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.043336 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 7833167a80c34f3e85bf9e79185ad84a (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.174261 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c24755244c47ba982f753b3f609119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.090865 6814 raft_consensus.cc:482] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.177168 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7833167a80c34f3e85bf9e79185ad84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.177076 6814 raft_consensus.cc:504] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.177999 6814 leader_election.cc:288] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:45.178436 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3873d0d3a0da4c7196d80148e86301f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.178625 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc0f45293fd48dfb1378a8c98b97dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.181062 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543c396fb7584cf3bcdcadf73cc8f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.181330 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827c062351e343b8b2fcc3811d5f305b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.185623 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d74dcf665da4dfd9116638b298913b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.188598 6193 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.125:46221, user_credentials={real_user=slave}} blocked reactor thread for 614856us
W20240503 02:12:45.103758 4746 consensus_peers.cc:485] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:45.189328 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 85934.9us
I20240503 02:12:45.193679 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:45.197121 4746 leader_election.cc:341] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.164819 5114 rpcz_store.cc:276] 0503 02:12:43.088760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.091040 (+ 2280us) service_pool.cc:225] Handling call
0503 02:12:45.164595 (+2073555us) ts_tablet_manager.cc:1120] Deleting tablet 192460518d49417cb8f03ac6ec971eef
0503 02:12:45.164604 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.164673 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991786}
I20240503 02:12:45.205631 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 890) took 2202 ms. Trace:
I20240503 02:12:45.205754 5114 rpcz_store.cc:276] 0503 02:12:43.003486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.063828 (+ 60342us) service_pool.cc:225] Handling call
0503 02:12:45.205492 (+2141664us) ts_tablet_manager.cc:1120] Deleting tablet b2560f50415444b8b2e497453a558d3a
0503 02:12:45.205504 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.205604 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993155}
I20240503 02:12:45.217182 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:45.221371 4746 leader_election.cc:341] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.221549 4746 leader_election.cc:302] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.221828 6814 raft_consensus.cc:2726] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.224738 5114 tablet_replica.cc:330] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:45.225083 5114 raft_consensus.cc:2229] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.225355 5114 raft_consensus.cc:2258] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.226322 5114 ts_tablet_manager.cc:1875] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.231073 6762 raft_consensus.cc:482] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.231309 6762 raft_consensus.cc:504] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.232242 6762 leader_election.cc:288] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:45.238577 4462 leader_election.cc:341] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:45.239360 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 38c24755244c47ba982f753b3f609119 (DEFAULT_TABLE table=default.Table_cf51dfd849f741bcb578d641aee31de4 [id=bc22ad4319f74d3b96f932858abed658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.239836 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c24755244c47ba982f753b3f609119. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.242115 4462 consensus_peers.cc:485] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607e49cde2fd4aab9f44921d3c3921a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.249517 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 7833167a80c34f3e85bf9e79185ad84a (DEFAULT_TABLE table=default.Table_f425f7b64daf435a8f9367ef0c1dbae1 [id=0efa136c1e6b4b8ca3ee3bd42ed66429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.250152 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7833167a80c34f3e85bf9e79185ad84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.259498 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 827c062351e343b8b2fcc3811d5f305b (DEFAULT_TABLE table=default.Table_6ce4d17d45734d5483fa95f40b455dd6 [id=913fd4417ab04b3a9100e2c8e574e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.260219 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827c062351e343b8b2fcc3811d5f305b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.992755 6769 raft_consensus.cc:504] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.273512 6769 leader_election.cc:288] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.274825 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4436ab9e7542f5b3aca930f6c5a758 (DEFAULT_TABLE table=default.Table_731eebc7b98e4b78832e9d33d32e0b5e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.275295 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4436ab9e7542f5b3aca930f6c5a758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.275377 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 43119471f91c4a1ba900f267c88db9a6 (table default.Table_5e542f394cce413f99f8012fcc9fba9a [id=aef6e606e6f34ed78039dd362329ce77]) successfully deleted
W20240503 02:12:45.275678 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9e4dabd6e05948b98f02515023e2aaa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e4dabd6e05948b98f02515023e2aaa1 already in progress: creating tablet
W20240503 02:12:45.275879 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b5f444b22112407ca02fe86ac50167a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f444b22112407ca02fe86ac50167a9 already in progress: creating tablet
W20240503 02:12:45.276067 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet a6de1b96c15c4b44809a03ef99e077ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6de1b96c15c4b44809a03ef99e077ba already in progress: creating tablet
W20240503 02:12:45.276336 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 192460518d49417cb8f03ac6ec971eef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 192460518d49417cb8f03ac6ec971eef already in progress: creating tablet
W20240503 02:12:45.276540 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet b2560f50415444b8b2e497453a558d3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2560f50415444b8b2e497453a558d3a already in progress: creating tablet
I20240503 02:12:45.277976 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6649f00b334a61bf1182e938d81d85 (DEFAULT_TABLE table=default.Table_54079b9b61d744c9aca36afd92498b47 [id=f602609c07e74751a09866bfc4ce7d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.278220 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6649f00b334a61bf1182e938d81d85 (DEFAULT_TABLE table=default.Table_54079b9b61d744c9aca36afd92498b47 [id=f602609c07e74751a09866bfc4ce7d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.278446 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6649f00b334a61bf1182e938d81d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.278614 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6649f00b334a61bf1182e938d81d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.281437 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 694b59d610264a6183a99cf0c76ba66e (DEFAULT_TABLE table=default.Table_ccb80b3d1e0a41e282ba3a1039347590 [id=b1e60750569846dcb9b4cddb6f7f16f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.281975 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694b59d610264a6183a99cf0c76ba66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.286154 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.286432 4811 raft_consensus.cc:2454] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.288034 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.288282 4811 raft_consensus.cc:2454] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.289093 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.289300 4808 raft_consensus.cc:2454] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:45.290314 4608 leader_election.cc:341] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 95681e67db1e4485876d530133e8a514
I20240503 02:12:45.290479 4608 leader_election.cc:302] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.290935 6698 raft_consensus.cc:2726] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:45.291126 6698 consensus_queue.cc:1035] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.297613 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4436ab9e7542f5b3aca930f6c5a758 (DEFAULT_TABLE table=default.Table_731eebc7b98e4b78832e9d33d32e0b5e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.298241 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4436ab9e7542f5b3aca930f6c5a758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.303635 6769 raft_consensus.cc:482] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.303812 6769 raft_consensus.cc:504] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.304034 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b149e66a0446eeade190dd7f5d2bec" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:45.304246 4672 raft_consensus.cc:3037] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.304773 6769 leader_election.cc:288] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.307157 4672 raft_consensus.cc:2454] T 53b149e66a0446eeade190dd7f5d2bec P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:45.310032 5105 tablet_bootstrap.cc:492] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:45.310259 5105 ts_tablet_manager.cc:1367] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.438s user 0.090s sys 0.134s
I20240503 02:12:45.315595 5103 tablet_bootstrap.cc:492] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.316607 5103 ts_tablet_manager.cc:1367] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.172s user 0.004s sys 0.000s
I20240503 02:12:45.318121 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 1dee8ce8aee3473084404d914340426e (DEFAULT_TABLE table=default.Table_131368ef1ddb4611aeb046287d0ac6b0 [id=83388680e07a40e494a60f10c0453d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.318702 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dee8ce8aee3473084404d914340426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.319234 4790 tablet_service.cc:1467] Processing CreateTablet for tablet 246fa4d9be0a4d6da4a1b26a188290c8 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.319679 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fa4d9be0a4d6da4a1b26a188290c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.319926 4780 tablet_service.cc:1467] Processing CreateTablet for tablet f461792d56444e21bf06d8ec47a94e98 (DEFAULT_TABLE table=default.Table_899f907d25f64238a341605f4dd1a34d [id=17b743385b3949c790a5cbf537c17f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.320015 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 94a609dac84a4305a2cbf8af56f0c055 (DEFAULT_TABLE table=default.Table_0d5505a598db4432af3d4cbe09a8aa3d [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.320288 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f461792d56444e21bf06d8ec47a94e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.322896 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a609dac84a4305a2cbf8af56f0c055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.325287 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 620223977a5f48209f6cc12b53915508 (DEFAULT_TABLE table=default.Table_c5703a4668b04accbd9b3f93427b31ab [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.325768 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620223977a5f48209f6cc12b53915508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.328835 5159 ts_tablet_manager.cc:1888] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:45.334564 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 543c396fb7584cf3bcdcadf73cc8f5e5 (DEFAULT_TABLE table=default.Table_103a11fe81534cdbb336f9b1b18bfb33 [id=6760740172574bc6b9322db6e9d52184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.335619 5103 raft_consensus.cc:348] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.341672 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543c396fb7584cf3bcdcadf73cc8f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.318615 4495 tablet_service.cc:1467] Processing CreateTablet for tablet ea9b188f504a47a9b6f26c999e10d508 (DEFAULT_TABLE table=default.Table_e725732066ff4ce69168444e3457b10c [id=a0f3401c46674942b78fae8e9179bea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.343587 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9b188f504a47a9b6f26c999e10d508. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.334745 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 43640.9us
W20240503 02:12:45.344492 4608 consensus_peers.cc:485] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.346065 5106 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.346210 5106 ts_tablet_manager.cc:1367] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.340s user 0.011s sys 0.042s
I20240503 02:12:45.347508 5106 raft_consensus.cc:348] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.347697 5106 raft_consensus.cc:374] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.347740 5106 raft_consensus.cc:729] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.348143 5106 consensus_queue.cc:260] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.348749 5106 ts_tablet_manager.cc:1398] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:45.318946 4492 tablet_service.cc:1467] Processing CreateTablet for tablet f461792d56444e21bf06d8ec47a94e98 (DEFAULT_TABLE table=default.Table_899f907d25f64238a341605f4dd1a34d [id=17b743385b3949c790a5cbf537c17f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.349031 5106 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.349359 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f461792d56444e21bf06d8ec47a94e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.350456 5106 tablet_bootstrap.cc:654] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.319226 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 94a609dac84a4305a2cbf8af56f0c055 (DEFAULT_TABLE table=default.Table_0d5505a598db4432af3d4cbe09a8aa3d [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.353401 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a609dac84a4305a2cbf8af56f0c055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.319550 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 04b848eb5059458c8e264e82f47ab818 (DEFAULT_TABLE table=default.Table_5be4b47f18164bf1882c0d2e917e43ff [id=cfa831af34f34546a100b29105d97899]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.322139 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.321398 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.325592 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.326272 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 620223977a5f48209f6cc12b53915508 (DEFAULT_TABLE table=default.Table_c5703a4668b04accbd9b3f93427b31ab [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.326620 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 246fa4d9be0a4d6da4a1b26a188290c8 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.358976 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b848eb5059458c8e264e82f47ab818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.327092 4462 leader_election.cc:302] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:45.318055 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 1dee8ce8aee3473084404d914340426e (DEFAULT_TABLE table=default.Table_131368ef1ddb4611aeb046287d0ac6b0 [id=83388680e07a40e494a60f10c0453d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.359215 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fa4d9be0a4d6da4a1b26a188290c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.359418 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dee8ce8aee3473084404d914340426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.365092 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620223977a5f48209f6cc12b53915508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.365135 6769 raft_consensus.cc:2781] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.365460 6769 raft_consensus.cc:686] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:45.365849 6769 consensus_queue.cc:237] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:45.369429 4608 consensus_peers.cc:485] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f9994ca5aea48d3bc25c097dddba3c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.369721 4608 leader_election.cc:302] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:45.369998 6762 raft_consensus.cc:2781] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.370103 6762 raft_consensus.cc:482] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.370216 6762 raft_consensus.cc:3037] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:45.371939 4462 leader_election.cc:341] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 53b149e66a0446eeade190dd7f5d2bec
I20240503 02:12:45.373167 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.374408 4608 leader_election.cc:302] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:45.374678 6792 raft_consensus.cc:2781] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.374751 6792 raft_consensus.cc:482] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.374809 6792 raft_consensus.cc:3037] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.375250 4608 leader_election.cc:302] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:45.377552 5103 raft_consensus.cc:374] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.342262 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe121a1b274d91a0a91d45d6642144 (DEFAULT_TABLE table=default.Table_cd9386a1ac724110944b2963851c7442 [id=ce4674ca63094d70bf4865814e43c232]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.378679 5103 raft_consensus.cc:729] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.318933 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8d0494ee4bd746069bfaf39bfb46687d (table default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]) successfully deleted
W20240503 02:12:45.379106 4259 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 60139.3us
I20240503 02:12:45.379626 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe121a1b274d91a0a91d45d6642144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.385388 4273 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_8c5ce4d70c734f70a52c3e3267097103 [id=a401eebd3ac541e19b6f3f310135a4da] alter complete (version 3)
I20240503 02:12:45.388705 4273 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 289300b955dd41c09d2fc348224bb9f4 (table default.Table_b145550ee4dc4b6a8f00b076f2ac72a5 [id=2817ff73927a4593bddef5408db5cc49]). Expected version 1 got 0
I20240503 02:12:45.389256 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet db6fa251094842b9a629fdb67e20690b
I20240503 02:12:45.390156 4270 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5c3f0c84a0db4405ad8fed6849923e96" } modify_external_catalogs: true
I20240503 02:12:45.390370 4270 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5c3f0c84a0db4405ad8fed6849923e96" } modify_external_catalogs: true
I20240503 02:12:45.329439 4776 tablet_service.cc:1467] Processing CreateTablet for tablet 04b848eb5059458c8e264e82f47ab818 (DEFAULT_TABLE table=default.Table_5be4b47f18164bf1882c0d2e917e43ff [id=cfa831af34f34546a100b29105d97899]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.387483 6738 raft_consensus.cc:482] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.392304 6738 raft_consensus.cc:504] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.392402 4274 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5e542f394cce413f99f8012fcc9fba9a" } modify_external_catalogs: true
I20240503 02:12:45.392627 4274 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5e542f394cce413f99f8012fcc9fba9a" } modify_external_catalogs: true
I20240503 02:12:45.393071 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b848eb5059458c8e264e82f47ab818. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.394430 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4e84a639b2403082b737d073f00ad7 consensus state has no leader
I20240503 02:12:45.394980 4269 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_db309e0e41744d558fd04a5e5766718f" } new_table_name: "default.Table_8e63c958cc71458a9473fd19b2898911" modify_external_catalogs: true
I20240503 02:12:45.395651 5159 log.cc:1196] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1436dd3df7284f9bb379236eb9d76836
I20240503 02:12:45.396021 5159 ts_tablet_manager.cc:1909] T 1436dd3df7284f9bb379236eb9d76836 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:45.396759 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efa7c0c5fc614c16985008b8fba54bab consensus state has no leader
I20240503 02:12:45.397138 4268 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_afc821e2c5794ea190f4aec89a0f4676" } new_table_name: "default.Table_f74e9bf09512430ca589ae5c79060999" modify_external_catalogs: true
I20240503 02:12:45.398627 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet 7c19263f06ec41d7a6fe6a8df7f88f8f from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:45.399183 4266 master_service.cc:946] ReplaceTablet: received request to replace tablet b7d0640cea2a482e9974376ba479163b from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:45.399753 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_c9320e1ce9bc491faf424b7db2850839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:45.396895 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 906) took 3296 ms. Trace:
W20240503 02:12:45.400329 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9320e1ce9bc491faf424b7db2850839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.400290 5159 rpcz_store.cc:276] 0503 02:12:42.100263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.100331 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:45.159378 (+3059047us) ts_tablet_manager.cc:1120] Deleting tablet 1436dd3df7284f9bb379236eb9d76836
0503 02:12:45.159392 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.210749 (+ 51357us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.328718 (+117969us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.396873 (+ 68155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3058782}
I20240503 02:12:45.400635 5159 tablet_replica.cc:330] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:45.400813 5159 raft_consensus.cc:2229] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.400980 5159 raft_consensus.cc:2258] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.401196 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet ffef15b7e51840fea91b5349a1724cc5
I20240503 02:12:45.401518 5159 ts_tablet_manager.cc:1875] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.402012 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d6cf353d37364d2098437daadf566b51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.402729 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6cf353d37364d2098437daadf566b51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.405725 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet a74f4816cad04a9eb5934aaba86c2fbc (table default.Table_6681572bb5e5469bb236a2eb7975fd0a [id=94ad6cc60edf4de887ceaf683a16b4f8]) successfully deleted
W20240503 02:12:45.406033 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 44707b6439b141b7b63f49f34f25b37a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44707b6439b141b7b63f49f34f25b37a already in progress: creating tablet
W20240503 02:12:45.406278 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ab093285c39b4ce7a0bb3de9d016a918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab093285c39b4ce7a0bb3de9d016a918 already in progress: creating tablet
W20240503 02:12:45.406534 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4fb59bbf7b68431c86d88f27ef2528b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fb59bbf7b68431c86d88f27ef2528b0 already in progress: creating tablet
W20240503 02:12:45.406737 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 95c7e4a2954e4161b9acc029873f923b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c7e4a2954e4161b9acc029873f923b already in progress: creating tablet
W20240503 02:12:45.406952 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 41531ddcbf66465094d8f06e1116b3be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41531ddcbf66465094d8f06e1116b3be already in progress: creating tablet
I20240503 02:12:45.425755 6739 raft_consensus.cc:482] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.425935 6739 raft_consensus.cc:504] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.426585 6739 leader_election.cc:288] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:45.430931 4273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_f028f0dfeea1433ca95764504d64c85e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.431710 4273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f028f0dfeea1433ca95764504d64c85e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.434862 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet db6fa251094842b9a629fdb67e20690b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.434916 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet ffef15b7e51840fea91b5349a1724cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.447446 4783 tablet_service.cc:1467] Processing CreateTablet for tablet ea9b188f504a47a9b6f26c999e10d508 (DEFAULT_TABLE table=default.Table_e725732066ff4ce69168444e3457b10c [id=a0f3401c46674942b78fae8e9179bea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.454612 4267 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 92126108432e4450ba30cfde8ca8618f
I20240503 02:12:45.455485 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_23b17ecf55e548ff9ab71cd49b7d6fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.456138 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23b17ecf55e548ff9ab71cd49b7d6fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.384955 6811 consensus_queue.cc:1035] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:45.457206 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f3f3967a2604dc2aa6a0f6f867fc298 consensus state has no leader
I20240503 02:12:45.232846 5114 ts_tablet_manager.cc:1888] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:45.457684 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_997cf815690744c3a6c9732ff0802a5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:45.457850 5114 log.cc:1196] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/8195767d80384a6c9a4690292f6776de
I20240503 02:12:45.458261 5114 ts_tablet_manager.cc:1909] T 8195767d80384a6c9a4690292f6776de P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:45.458328 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_997cf815690744c3a6c9732ff0802a5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.459097 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 901) took 2205 ms. Trace:
I20240503 02:12:45.439795 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.445802 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet ffef15b7e51840fea91b5349a1724cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.445991 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet db6fa251094842b9a629fdb67e20690b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.459264 5114 rpcz_store.cc:276] 0503 02:12:43.253466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.254650 (+ 1184us) service_pool.cc:225] Handling call
0503 02:12:45.224668 (+1970018us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:45.224684 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.229653 (+ 4969us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.232722 (+ 3069us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.459074 (+226352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969812}
I20240503 02:12:45.461480 5114 tablet_replica.cc:330] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:45.462893 5114 raft_consensus.cc:2229] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:45.463280 5114 raft_consensus.cc:2258] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.463959 5114 ts_tablet_manager.cc:1875] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.463984 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3478033c4ed94268a2436a2942c27a68" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.465134 4271 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_29b8558332544b49aefeb049f9452eb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.465818 4271 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29b8558332544b49aefeb049f9452eb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:45.469722 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 56211.3us
I20240503 02:12:45.471180 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 8195767d80384a6c9a4690292f6776de (table default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]) successfully deleted
I20240503 02:12:45.471580 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1436dd3df7284f9bb379236eb9d76836 (table default.Table_7fe2fd4286f4411aad562211e4bd77ae [id=2073875662ee44ca99e5837ba4ace7ca]) successfully deleted
I20240503 02:12:45.478359 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 92126108432e4450ba30cfde8ca8618f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.392081 6829 raft_consensus.cc:2781] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.486416 6829 raft_consensus.cc:482] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.486572 6829 raft_consensus.cc:3037] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.482134 5103 consensus_queue.cc:260] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.487839 5103 ts_tablet_manager.cc:1398] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.153s user 0.002s sys 0.000s
I20240503 02:12:45.459918 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289300b955dd41c09d2fc348224bb9f4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.489377 6829 raft_consensus.cc:504] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.490108 6829 leader_election.cc:288] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.393631 6831 tablet.cc:1722] T 289300b955dd41c09d2fc348224bb9f4 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:45.398703 6762 raft_consensus.cc:504] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.491025 6762 leader_election.cc:288] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.401448 6792 raft_consensus.cc:504] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.491569 6792 leader_election.cc:288] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.416711 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6649f00b334a61bf1182e938d81d85 (DEFAULT_TABLE table=default.Table_54079b9b61d744c9aca36afd92498b47 [id=f602609c07e74751a09866bfc4ce7d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.491984 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6649f00b334a61bf1182e938d81d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.494764 5115 raft_consensus.cc:2258] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.495442 5115 ts_tablet_manager.cc:1875] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.417308 4638 tablet_service.cc:1467] Processing CreateTablet for tablet ea9b188f504a47a9b6f26c999e10d508 (DEFAULT_TABLE table=default.Table_e725732066ff4ce69168444e3457b10c [id=a0f3401c46674942b78fae8e9179bea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.497062 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9b188f504a47a9b6f26c999e10d508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.417980 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 1dee8ce8aee3473084404d914340426e (DEFAULT_TABLE table=default.Table_131368ef1ddb4611aeb046287d0ac6b0 [id=83388680e07a40e494a60f10c0453d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.418244 5106 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.418452 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 04b848eb5059458c8e264e82f47ab818 (DEFAULT_TABLE table=default.Table_5be4b47f18164bf1882c0d2e917e43ff [id=cfa831af34f34546a100b29105d97899]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.501243 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dee8ce8aee3473084404d914340426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.418934 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 246fa4d9be0a4d6da4a1b26a188290c8 (DEFAULT_TABLE table=default.Table_a0e38fafc9ce4b6d89729634d5b0243d [id=120b31b55d6a4619a21422102fcf142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.501545 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fa4d9be0a4d6da4a1b26a188290c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.419342 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 94a609dac84a4305a2cbf8af56f0c055 (DEFAULT_TABLE table=default.Table_0d5505a598db4432af3d4cbe09a8aa3d [id=dc092cb85ccf497e8e1bea1c3b0fd423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.505251 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a609dac84a4305a2cbf8af56f0c055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.505360 5103 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.507054 5103 tablet_bootstrap.cc:654] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.419852 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet db6fa251094842b9a629fdb67e20690b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:44 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.420126 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet ffef15b7e51840fea91b5349a1724cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.420630 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 620223977a5f48209f6cc12b53915508 (DEFAULT_TABLE table=default.Table_c5703a4668b04accbd9b3f93427b31ab [id=838900269aa04721a4033aab3345c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.512583 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620223977a5f48209f6cc12b53915508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.512583 6738 leader_election.cc:288] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:45.513098 5106 ts_tablet_manager.cc:1367] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.000s
I20240503 02:12:45.513432 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b848eb5059458c8e264e82f47ab818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.420959 4650 tablet_service.cc:1467] Processing CreateTablet for tablet f461792d56444e21bf06d8ec47a94e98 (DEFAULT_TABLE table=default.Table_899f907d25f64238a341605f4dd1a34d [id=17b743385b3949c790a5cbf537c17f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.421134 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4436ab9e7542f5b3aca930f6c5a758 (DEFAULT_TABLE table=default.Table_731eebc7b98e4b78832e9d33d32e0b5e [id=779c0d9ea5a6420d8ede93243a64261e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:45.437361 4608 leader_election.cc:341] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.517275 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f461792d56444e21bf06d8ec47a94e98. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:45.517277 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 79921.2us
W20240503 02:12:45.517553 4608 leader_election.cc:341] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.517599 5106 raft_consensus.cc:348] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.517812 5106 raft_consensus.cc:374] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:45.517841 4608 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.438071 6833 consensus_queue.cc:1035] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.462265 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3478033c4ed94268a2436a2942c27a68" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.520220 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4436ab9e7542f5b3aca930f6c5a758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.526017 6830 raft_consensus.cc:482] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.526211 6830 raft_consensus.cc:504] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.526783 6830 leader_election.cc:288] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.527508 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 607e49cde2fd4aab9f44921d3c3921a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.527714 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.527871 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet a74f4816cad04a9eb5934aaba86c2fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:21 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.527882 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 92126108432e4450ba30cfde8ca8618f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:45.528216 4608 consensus_peers.cc:485] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88b173d78ba74ede801b12cccbeef6b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.529060 6792 consensus_queue.cc:1035] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.530855 4518 raft_consensus.cc:3037] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.533756 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:45.535741 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:45.536499 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:45.517856 5106 raft_consensus.cc:729] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.538522 5106 consensus_queue.cc:260] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.539085 5106 ts_tablet_manager.cc:1398] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240503 02:12:45.541229 5106 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.541666 4274 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 368aabce19984d03aa7116c8df990ef1
W20240503 02:12:45.545303 4268 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47ede57814354d54b3780f1d0ddaa684 consensus state has no leader
I20240503 02:12:45.545749 4268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_fbe14bb502bb465a95363bfcbbded773"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.546409 4268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbe14bb502bb465a95363bfcbbded773 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.547922 5106 tablet_bootstrap.cc:654] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.548621 4267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_5bce1cbd35204706ab1c70da3a37d8dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.549197 4269 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61f2534d4e27419e9a3272de1abc4373 consensus state has no leader
I20240503 02:12:45.549425 4269 master_service.cc:946] ReplaceTablet: received request to replace tablet 972cd318ebed41c0ab1b8705f52bdaa4 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:45.550388 4746 leader_election.cc:341] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.550719 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 20b6a18be64049049e1a083bf4af51ba from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:45.551942 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b149e66a0446eeade190dd7f5d2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.553242 4270 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 1cdfaff4b319415db858f06c47829933
I20240503 02:12:45.553484 4462 leader_election.cc:302] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:45.554031 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d607da9689c14e51add9178a48b35913" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:45.554216 4608 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.554450 4608 leader_election.cc:341] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.554594 4608 leader_election.cc:341] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.557334 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 92126108432e4450ba30cfde8ca8618f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
W20240503 02:12:45.557965 4267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bce1cbd35204706ab1c70da3a37d8dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.558305 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.558720 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73965590a0ea4bffae6e99fa1367fad0" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.558882 4807 raft_consensus.cc:3037] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.559163 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.559322 4810 raft_consensus.cc:3037] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:45.559782 4608 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:45.559885 4608 leader_election.cc:302] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.560216 6841 raft_consensus.cc:2726] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.561771 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d607da9689c14e51add9178a48b35913" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.561959 4810 raft_consensus.cc:2454] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.563057 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:45.563611 4608 leader_election.cc:341] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.544816 4274 catalog_manager.cc:5484] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65). New cstate: current_term: 1 leader_uuid: "f4f9f079230a468f90e5e3a58f254b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:45.563819 4608 leader_election.cc:302] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.564188 4274 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 289300b955dd41c09d2fc348224bb9f4 (table default.Table_b145550ee4dc4b6a8f00b076f2ac72a5 [id=2817ff73927a4593bddef5408db5cc49]). Expected version 1 got 0
I20240503 02:12:45.564486 6829 raft_consensus.cc:2781] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.564600 6829 raft_consensus.cc:686] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.564595 5106 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.564821 5106 ts_tablet_manager.cc:1367] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240503 02:12:45.564810 6829 consensus_queue.cc:237] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:45.565814 4746 leader_election.cc:341] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.565980 4746 leader_election.cc:302] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:45.566617 4746 leader_election.cc:341] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.567085 4746 leader_election.cc:341] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.567157 4462 consensus_peers.cc:485] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b38d63d2cc8c4f54ba43cdf8cde1841f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.567185 4746 leader_election.cc:302] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.567451 6739 raft_consensus.cc:2726] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.567644 6814 raft_consensus.cc:2726] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.570780 4807 raft_consensus.cc:2454] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.571579 5106 raft_consensus.cc:348] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.571828 5106 raft_consensus.cc:374] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.572026 5106 raft_consensus.cc:729] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.572690 5106 consensus_queue.cc:260] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.573519 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.573824 4812 raft_consensus.cc:2454] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.576880 5106 ts_tablet_manager.cc:1398] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:12:45.577178 5106 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.578552 5106 tablet_bootstrap.cc:654] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.580951 5106 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.581686 5106 ts_tablet_manager.cc:1367] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:12:45.582618 5106 raft_consensus.cc:348] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.567623 6739 consensus_queue.cc:1035] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 02:12:45.586143 5106 raft_consensus.cc:374] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.586448 5106 raft_consensus.cc:729] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.588088 5106 consensus_queue.cc:260] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.590037 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 368aabce19984d03aa7116c8df990ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.592017 5106 ts_tablet_manager.cc:1398] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:12:45.592988 5106 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.594477 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdfaff4b319415db858f06c47829933 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.594981 5106 tablet_bootstrap.cc:654] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.601881 5103 tablet_bootstrap.cc:492] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.602569 5103 ts_tablet_manager.cc:1367] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20240503 02:12:45.605742 5103 raft_consensus.cc:348] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.607123 4608 leader_election.cc:302] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.607442 6841 raft_consensus.cc:2781] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.607566 6841 raft_consensus.cc:686] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.607573 4608 leader_election.cc:302] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.607827 6841 consensus_queue.cc:237] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.607981 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdfaff4b319415db858f06c47829933 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.608124 6699 raft_consensus.cc:2781] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.608199 6699 raft_consensus.cc:482] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.608258 6699 raft_consensus.cc:3037] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.611666 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_86543c68f82543d1bce59d92c6138b7b" } new_table_name: "default.Table_fb6461cb30dd4f8bb6089381b352f7f5" modify_external_catalogs: true
I20240503 02:12:45.613246 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_d2eeb6ba099049a1845ced64b6119f4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.613801 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2eeb6ba099049a1845ced64b6119f4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.615029 4270 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_8445a7ddbfe94d78a5489dd954b51d8b" } new_table_name: "default.Table_76d4f904da56406595b0e3da677e7cc8" modify_external_catalogs: true
I20240503 02:12:45.616434 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2b1d96d5b6c942f0914e000795c0386f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.616900 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b1d96d5b6c942f0914e000795c0386f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.602730 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdfaff4b319415db858f06c47829933 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.616480 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 368aabce19984d03aa7116c8df990ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.617589 5103 raft_consensus.cc:374] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.617688 5103 raft_consensus.cc:729] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.617946 5103 consensus_queue.cc:260] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.618157 4271 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_064299ed49e74e50a322f6cbf4878b98" } modify_external_catalogs: true
I20240503 02:12:45.618324 4271 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_064299ed49e74e50a322f6cbf4878b98" } modify_external_catalogs: true
I20240503 02:12:45.618604 5103 ts_tablet_manager.cc:1398] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 02:12:45.618958 5103 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.619550 5106 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.620353 5106 ts_tablet_manager.cc:1367] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240503 02:12:45.621331 5106 raft_consensus.cc:348] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.624023 5103 tablet_bootstrap.cc:654] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.622098 6699 raft_consensus.cc:504] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.625352 6699 leader_election.cc:288] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.628532 4273 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5b3b2e6e211c47338aa5ddca26669a20" } modify_external_catalogs: true
I20240503 02:12:45.628906 4273 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5b3b2e6e211c47338aa5ddca26669a20" } modify_external_catalogs: true
I20240503 02:12:45.626202 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:45.633682 4608 leader_election.cc:341] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.641268 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 368aabce19984d03aa7116c8df990ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.642167 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9b188f504a47a9b6f26c999e10d508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.646324 6619 raft_consensus.cc:2726] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:45.650388 4259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.17.126:43321, user_credentials={real_user=slave}} blocked reactor thread for 47847.6us
I20240503 02:12:45.657806 5114 ts_tablet_manager.cc:1888] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:45.624392 5106 raft_consensus.cc:374] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.658268 5106 raft_consensus.cc:729] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.658634 5106 consensus_queue.cc:260] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.659432 5106 ts_tablet_manager.cc:1398] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:12:45.659783 5106 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.662287 5106 tablet_bootstrap.cc:654] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.661015 4525 raft_consensus.cc:3037] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.665104 5114 log.cc:1196] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/2f862891b43a4845b723bfdb9a7da591
W20240503 02:12:45.665954 4723 tablet.cc:2369] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:45.642036 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b6e303e34a465caffed6c9e1e6bf74" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.662056 6857 tablet.cc:1722] T 289300b955dd41c09d2fc348224bb9f4 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:45.669495 6841 consensus_queue.cc:1035] T 1975c790bfa4449baa6d7b81509c5003 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.674495 5103 tablet_bootstrap.cc:492] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.668263 4812 raft_consensus.cc:3037] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.668776 5105 raft_consensus.cc:348] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.674919 5105 raft_consensus.cc:374] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.674973 5105 raft_consensus.cc:729] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.675213 5105 consensus_queue.cc:260] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.675808 5105 ts_tablet_manager.cc:1398] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20240503 02:12:45.676164 5103 ts_tablet_manager.cc:1367] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.006s
I20240503 02:12:45.677115 5103 raft_consensus.cc:348] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.677423 5103 raft_consensus.cc:374] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.677546 5103 raft_consensus.cc:729] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.677858 5103 consensus_queue.cc:260] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.685071 5103 ts_tablet_manager.cc:1398] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 02:12:45.686317 5103 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.689065 6841 raft_consensus.cc:482] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.689234 6841 raft_consensus.cc:504] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:44.374859 6238 rpcz_store.cc:276] 0503 02:12:37.264023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:37.264087 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:43.196909 (+5932822us) raft_consensus.cc:1457] Updating replica for 188 ops
0503 02:12:43.208689 (+ 11780us) raft_consensus.cc:1520] Early marking committed up to index 0
0503 02:12:43.208698 (+ 9us) raft_consensus.cc:1525] Triggering prepare for 188 ops
0503 02:12:43.209462 (+ 764us) write_op.cc:270] Start()
0503 02:12:43.209539 (+ 77us) write_op.cc:276] Timestamp: P: 1714702351604186 usec, L: 0
0503 02:12:43.380124 (+170585us) write_op.cc:270] Start()
0503 02:12:43.380177 (+ 53us) write_op.cc:276] Timestamp: P: 1714702351605925 usec, L: 0
0503 02:12:43.380465 (+ 288us) write_op.cc:270] Start()
0503 02:12:43.380504 (+ 39us) write_op.cc:276] Timestamp: P: 1714702351607699 usec, L: 0
0503 02:12:43.380729 (+ 225us) write_op.cc:270] Start()
0503 02:12:43.380752 (+ 23us) write_op.cc:276] Timestamp: P: 1714702351630734 usec, L: 0
0503 02:12:43.380907 (+ 155us) write_op.cc:270] Start()
0503 02:12:43.394202 (+ 13295us) write_op.cc:276] Timestamp: P: 1714702351632400 usec, L: 0
0503 02:12:43.394464 (+ 262us) write_op.cc:270] Start()
0503 02:12:43.394499 (+ 35us) write_op.cc:276] Timestamp: P: 1714702351666891 usec, L: 0
0503 02:12:43.394620 (+ 121us) write_op.cc:270] Start()
0503 02:12:43.394678 (+ 58us) write_op.cc:276] Timestamp: P: 1714702351667640 usec, L: 0
0503 02:12:43.394805 (+ 127us) write_op.cc:270] Start()
0503 02:12:43.394821 (+ 16us) write_op.cc:276] Timestamp: P: 1714702351668328 usec, L: 0
0503 02:12:43.394929 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.394944 (+ 15us) write_op.cc:276] Timestamp: P: 1714702352534140 usec, L: 0
0503 02:12:43.395041 (+ 97us) write_op.cc:270] Start()
0503 02:12:43.395057 (+ 16us) write_op.cc:276] Timestamp: P: 1714702352640608 usec, L: 0
0503 02:12:43.395200 (+ 143us) write_op.cc:270] Start()
0503 02:12:43.395219 (+ 19us) write_op.cc:276] Timestamp: P: 1714702352644418 usec, L: 0
0503 02:12:43.395360 (+ 141us) write_op.cc:270] Start()
0503 02:12:43.395383 (+ 23us) write_op.cc:276] Timestamp: P: 1714702352676226 usec, L: 0
0503 02:12:43.395528 (+ 145us) write_op.cc:270] Start()
0503 02:12:43.395606 (+ 78us) write_op.cc:276] Timestamp: P: 1714702352691229 usec, L: 0
0503 02:12:43.395738 (+ 132us) write_op.cc:270] Start()
0503 02:12:43.395757 (+ 19us) write_op.cc:276] Timestamp: P: 1714702352693687 usec, L: 0
0503 02:12:43.395852 (+ 95us) write_op.cc:270] Start()
0503 02:12:43.395868 (+ 16us) write_op.cc:276] Timestamp: P: 1714702352694321 usec, L: 0
0503 02:12:43.395977 (+ 109us) write_op.cc:270] Start()
0503 02:12:43.395993 (+ 16us) write_op.cc:276] Timestamp: P: 1714702352707029 usec, L: 0
0503 02:12:43.396109 (+ 116us) write_op.cc:270] Start()
0503 02:12:43.396132 (+ 23us) write_op.cc:276] Timestamp: P: 1714702352741755 usec, L: 0
0503 02:12:43.396240 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.396261 (+ 21us) write_op.cc:276] Timestamp: P: 1714702352742471 usec, L: 0
0503 02:12:43.396410 (+ 149us) write_op.cc:270] Start()
0503 02:12:43.396434 (+ 24us) write_op.cc:276] Timestamp: P: 1714702352743101 usec, L: 0
0503 02:12:43.396583 (+ 149us) write_op.cc:270] Start()
0503 02:12:43.396603 (+ 20us) write_op.cc:276] Timestamp: P: 1714702352743779 usec, L: 0
0503 02:12:43.396868 (+ 265us) write_op.cc:270] Start()
0503 02:12:43.396891 (+ 23us) write_op.cc:276] Timestamp: P: 1714702352762415 usec, L: 0
0503 02:12:43.433865 (+ 36974us) write_op.cc:270] Start()
0503 02:12:43.433935 (+ 70us) write_op.cc:276] Timestamp: P: 1714702352798166 usec, L: 0
0503 02:12:43.434134 (+ 199us) write_op.cc:270] Start()
0503 02:12:43.434158 (+ 24us) write_op.cc:276] Timestamp: P: 1714702352840047 usec, L: 0
0503 02:12:43.434302 (+ 144us) write_op.cc:270] Start()
0503 02:12:43.434326 (+ 24us) write_op.cc:276] Timestamp: P: 1714702352842359 usec, L: 0
0503 02:12:43.434561 (+ 235us) write_op.cc:270] Start()
0503 02:12:43.434590 (+ 29us) write_op.cc:276] Timestamp: P: 1714702352843551 usec, L: 0
0503 02:12:43.435234 (+ 644us) write_op.cc:270] Start()
0503 02:12:43.435258 (+ 24us) write_op.cc:276] Timestamp: P: 1714702352845082 usec, L: 0
0503 02:12:43.435401 (+ 143us) write_op.cc:270] Start()
0503 02:12:43.435421 (+ 20us) write_op.cc:276] Timestamp: P: 1714702353016767 usec, L: 0
0503 02:12:43.435546 (+ 125us) write_op.cc:270] Start()
0503 02:12:43.435563 (+ 17us) write_op.cc:276] Timestamp: P: 1714702353019495 usec, L: 0
0503 02:12:43.440152 (+ 4589us) write_op.cc:270] Start()
0503 02:12:43.440184 (+ 32us) write_op.cc:276] Timestamp: P: 1714702353028014 usec, L: 0
0503 02:12:43.440343 (+ 159us) write_op.cc:270] Start()
0503 02:12:43.440363 (+ 20us) write_op.cc:276] Timestamp: P: 1714702353086539 usec, L: 0
0503 02:12:43.440489 (+ 126us) write_op.cc:270] Start()
0503 02:12:43.440508 (+ 19us) write_op.cc:276] Timestamp: P: 1714702353091528 usec, L: 0
0503 02:12:43.440621 (+ 113us) write_op.cc:270] Start()
0503 02:12:43.440640 (+ 19us) write_op.cc:276] Timestamp: P: 1714702353093288 usec, L: 0
0503 02:12:43.440830 (+ 190us) write_op.cc:270] Start()
0503 02:12:43.440849 (+ 19us) write_op.cc:276] Timestamp: P: 1714702353110525 usec, L: 0
0503 02:12:43.441000 (+ 151us) write_op.cc:270] Start()
0503 02:12:43.441022 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353121232 usec, L: 0
0503 02:12:43.441146 (+ 124us) write_op.cc:270] Start()
0503 02:12:43.441163 (+ 17us) write_op.cc:276] Timestamp: P: 1714702353127191 usec, L: 0
0503 02:12:43.470254 (+ 29091us) write_op.cc:270] Start()
0503 02:12:43.470444 (+ 190us) write_op.cc:276] Timestamp: P: 1714702353129757 usec, L: 0
0503 02:12:43.470759 (+ 315us) write_op.cc:270] Start()
0503 02:12:43.470806 (+ 47us) write_op.cc:276] Timestamp: P: 1714702353137089 usec, L: 0
0503 02:12:43.471035 (+ 229us) write_op.cc:270] Start()
0503 02:12:43.471061 (+ 26us) write_op.cc:276] Timestamp: P: 1714702353139183 usec, L: 0
0503 02:12:43.471180 (+ 119us) write_op.cc:270] Start()
0503 02:12:43.471196 (+ 16us) write_op.cc:276] Timestamp: P: 1714702353208415 usec, L: 0
0503 02:12:43.471310 (+ 114us) write_op.cc:270] Start()
0503 02:12:43.471328 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353209846 usec, L: 0
0503 02:12:43.471448 (+ 120us) write_op.cc:270] Start()
0503 02:12:43.471473 (+ 25us) write_op.cc:276] Timestamp: P: 1714702353211470 usec, L: 0
0503 02:12:43.471603 (+ 130us) write_op.cc:270] Start()
0503 02:12:43.471621 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353213776 usec, L: 0
0503 02:12:43.471766 (+ 145us) write_op.cc:270] Start()
0503 02:12:43.471786 (+ 20us) write_op.cc:276] Timestamp: P: 1714702353216434 usec, L: 0
0503 02:12:43.471898 (+ 112us) write_op.cc:270] Start()
0503 02:12:43.471918 (+ 20us) write_op.cc:276] Timestamp: P: 1714702353217089 usec, L: 0
0503 02:12:43.472087 (+ 169us) write_op.cc:270] Start()
0503 02:12:43.472113 (+ 26us) write_op.cc:276] Timestamp: P: 1714702353228550 usec, L: 0
0503 02:12:43.472245 (+ 132us) write_op.cc:270] Start()
0503 02:12:43.472264 (+ 19us) write_op.cc:276] Timestamp: P: 1714702353240350 usec, L: 0
0503 02:12:43.472439 (+ 175us) write_op.cc:270] Start()
0503 02:12:43.472474 (+ 35us) write_op.cc:276] Timestamp: P: 1714702353241102 usec, L: 0
0503 02:12:43.472650 (+ 176us) write_op.cc:270] Start()
0503 02:12:43.472671 (+ 21us) write_op.cc:276] Timestamp: P: 1714702353291800 usec, L: 0
0503 02:12:43.472819 (+ 148us) write_op.cc:270] Start()
0503 02:12:43.472847 (+ 28us) write_op.cc:276] Timestamp: P: 1714702353292848 usec, L: 0
0503 02:12:43.527006 (+ 54159us) write_op.cc:270] Start()
0503 02:12:43.527046 (+ 40us) write_op.cc:276] Timestamp: P: 1714702353295490 usec, L: 0
0503 02:12:43.538817 (+ 11771us) write_op.cc:270] Start()
0503 02:12:43.538862 (+ 45us) write_op.cc:276] Timestamp: P: 1714702353299905 usec, L: 0
0503 02:12:43.539075 (+ 213us) write_op.cc:270] Start()
0503 02:12:43.539097 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353302388 usec, L: 0
0503 02:12:43.543019 (+ 3922us) write_op.cc:270] Start()
0503 02:12:43.543053 (+ 34us) write_op.cc:276] Timestamp: P: 1714702353346980 usec, L: 0
0503 02:12:43.543240 (+ 187us) write_op.cc:270] Start()
0503 02:12:43.543265 (+ 25us) write_op.cc:276] Timestamp: P: 1714702353354884 usec, L: 0
0503 02:12:43.545502 (+ 2237us) write_op.cc:270] Start()
0503 02:12:43.545543 (+ 41us) write_op.cc:276] Timestamp: P: 1714702353394642 usec, L: 0
0503 02:12:43.545820 (+ 277us) write_op.cc:270] Start()
0503 02:12:43.545846 (+ 26us) write_op.cc:276] Timestamp: P: 1714702353416811 usec, L: 0
0503 02:12:43.546034 (+ 188us) write_op.cc:270] Start()
0503 02:12:43.546059 (+ 25us) write_op.cc:276] Timestamp: P: 1714702353437225 usec, L: 0
0503 02:12:43.546193 (+ 134us) write_op.cc:270] Start()
0503 02:12:43.546211 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353437883 usec, L: 0
0503 02:12:43.546337 (+ 126us) write_op.cc:270] Start()
0503 02:12:43.546358 (+ 21us) write_op.cc:276] Timestamp: P: 1714702353438402 usec, L: 0
0503 02:12:43.546507 (+ 149us) write_op.cc:270] Start()
0503 02:12:43.546552 (+ 45us) write_op.cc:276] Timestamp: P: 1714702353439181 usec, L: 0
0503 02:12:43.546695 (+ 143us) write_op.cc:270] Start()
0503 02:12:43.546715 (+ 20us) write_op.cc:276] Timestamp: P: 1714702353439783 usec, L: 0
0503 02:12:43.546850 (+ 135us) write_op.cc:270] Start()
0503 02:12:43.546874 (+ 24us) write_op.cc:276] Timestamp: P: 1714702353442636 usec, L: 0
0503 02:12:43.546989 (+ 115us) write_op.cc:270] Start()
0503 02:12:43.547011 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353443229 usec, L: 0
0503 02:12:43.547138 (+ 127us) write_op.cc:270] Start()
0503 02:12:43.547156 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353508856 usec, L: 0
0503 02:12:43.547298 (+ 142us) write_op.cc:270] Start()
0503 02:12:43.547316 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353509835 usec, L: 0
0503 02:12:43.547412 (+ 96us) write_op.cc:270] Start()
0503 02:12:43.547428 (+ 16us) write_op.cc:276] Timestamp: P: 1714702353510697 usec, L: 0
0503 02:12:43.547534 (+ 106us) write_op.cc:270] Start()
0503 02:12:43.547551 (+ 17us) write_op.cc:276] Timestamp: P: 1714702353511346 usec, L: 0
0503 02:12:43.547672 (+ 121us) write_op.cc:270] Start()
0503 02:12:43.547690 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353511950 usec, L: 0
0503 02:12:43.547823 (+ 133us) write_op.cc:270] Start()
0503 02:12:43.547842 (+ 19us) write_op.cc:276] Timestamp: P: 1714702353550523 usec, L: 0
0503 02:12:43.548006 (+ 164us) write_op.cc:270] Start()
0503 02:12:43.548030 (+ 24us) write_op.cc:276] Timestamp: P: 1714702353551218 usec, L: 0
0503 02:12:43.548173 (+ 143us) write_op.cc:270] Start()
0503 02:12:43.548196 (+ 23us) write_op.cc:276] Timestamp: P: 1714702353551810 usec, L: 0
0503 02:12:43.548344 (+ 148us) write_op.cc:270] Start()
0503 02:12:43.548368 (+ 24us) write_op.cc:276] Timestamp: P: 1714702353552347 usec, L: 0
0503 02:12:43.548511 (+ 143us) write_op.cc:270] Start()
0503 02:12:43.548533 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353681621 usec, L: 0
0503 02:12:43.548669 (+ 136us) write_op.cc:270] Start()
0503 02:12:43.548690 (+ 21us) write_op.cc:276] Timestamp: P: 1714702353696423 usec, L: 0
0503 02:12:43.548827 (+ 137us) write_op.cc:270] Start()
0503 02:12:43.548849 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353697375 usec, L: 0
0503 02:12:43.642149 (+ 93300us) write_op.cc:270] Start()
0503 02:12:43.642195 (+ 46us) write_op.cc:276] Timestamp: P: 1714702353698119 usec, L: 0
0503 02:12:43.642479 (+ 284us) write_op.cc:270] Start()
0503 02:12:43.642507 (+ 28us) write_op.cc:276] Timestamp: P: 1714702353699108 usec, L: 0
0503 02:12:43.642675 (+ 168us) write_op.cc:270] Start()
0503 02:12:43.642696 (+ 21us) write_op.cc:276] Timestamp: P: 1714702353715588 usec, L: 0
0503 02:12:43.642851 (+ 155us) write_op.cc:270] Start()
0503 02:12:43.642897 (+ 46us) write_op.cc:276] Timestamp: P: 1714702353773910 usec, L: 0
0503 02:12:43.643090 (+ 193us) write_op.cc:270] Start()
0503 02:12:43.643114 (+ 24us) write_op.cc:276] Timestamp: P: 1714702353774913 usec, L: 0
0503 02:12:43.643281 (+ 167us) write_op.cc:270] Start()
0503 02:12:43.643303 (+ 22us) write_op.cc:276] Timestamp: P: 1714702353775757 usec, L: 0
0503 02:12:43.643435 (+ 132us) write_op.cc:270] Start()
0503 02:12:43.773566 (+130131us) write_op.cc:276] Timestamp: P: 1714702353776390 usec, L: 0
0503 02:12:43.781524 (+ 7958us) write_op.cc:270] Start()
0503 02:12:43.781567 (+ 43us) write_op.cc:276] Timestamp: P: 1714702353777022 usec, L: 0
0503 02:12:43.781737 (+ 170us) write_op.cc:270] Start()
0503 02:12:43.781762 (+ 25us) write_op.cc:276] Timestamp: P: 1714702353847317 usec, L: 0
0503 02:12:43.781900 (+ 138us) write_op.cc:270] Start()
0503 02:12:43.781918 (+ 18us) write_op.cc:276] Timestamp: P: 1714702353848079 usec, L: 0
0503 02:12:43.782039 (+ 121us) write_op.cc:270] Start()
0503 02:12:43.782055 (+ 16us) write_op.cc:276] Timestamp: P: 1714702353852316 usec, L: 0
0503 02:12:43.782163 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.782178 (+ 15us) write_op.cc:276] Timestamp: P: 1714702353862604 usec, L: 0
0503 02:12:43.782278 (+ 100us) write_op.cc:270] Start()
0503 02:12:43.782293 (+ 15us) write_op.cc:276] Timestamp: P: 1714702354116047 usec, L: 0
0503 02:12:43.782399 (+ 106us) write_op.cc:270] Start()
0503 02:12:43.782416 (+ 17us) write_op.cc:276] Timestamp: P: 1714702354120369 usec, L: 0
0503 02:12:43.782512 (+ 96us) write_op.cc:270] Start()
0503 02:12:43.782527 (+ 15us) write_op.cc:276] Timestamp: P: 1714702354121109 usec, L: 0
0503 02:12:43.782627 (+ 100us) write_op.cc:270] Start()
0503 02:12:43.782648 (+ 21us) write_op.cc:276] Timestamp: P: 1714702354121663 usec, L: 0
0503 02:12:43.782787 (+ 139us) write_op.cc:270] Start()
0503 02:12:43.782805 (+ 18us) write_op.cc:276] Timestamp: P: 1714702354123758 usec, L: 0
0503 02:12:43.782928 (+ 123us) write_op.cc:270] Start()
0503 02:12:43.782948 (+ 20us) write_op.cc:276] Timestamp: P: 1714702354448146 usec, L: 0
0503 02:12:43.783086 (+ 138us) write_op.cc:270] Start()
0503 02:12:43.783108 (+ 22us) write_op.cc:276] Timestamp: P: 1714702354449092 usec, L: 0
0503 02:12:43.783248 (+ 140us) write_op.cc:270] Start()
0503 02:12:43.783267 (+ 19us) write_op.cc:276] Timestamp: P: 1714702354465199 usec, L: 0
0503 02:12:43.783403 (+ 136us) write_op.cc:270] Start()
0503 02:12:43.783426 (+ 23us) write_op.cc:276] Timestamp: P: 1714702354585784 usec, L: 0
0503 02:12:43.783543 (+ 117us) write_op.cc:270] Start()
0503 02:12:43.783560 (+ 17us) write_op.cc:276] Timestamp: P: 1714702354594835 usec, L: 0
0503 02:12:43.783681 (+ 121us) write_op.cc:270] Start()
0503 02:12:43.783699 (+ 18us) write_op.cc:276] Timestamp: P: 1714702354595641 usec, L: 0
0503 02:12:43.783806 (+ 107us) write_op.cc:270] Start()
0503 02:12:43.783822 (+ 16us) write_op.cc:276] Timestamp: P: 1714702354598529 usec, L: 0
0503 02:12:43.783953 (+ 131us) write_op.cc:270] Start()
0503 02:12:43.783973 (+ 20us) write_op.cc:276] Timestamp: P: 1714702354663648 usec, L: 0
0503 02:12:43.784113 (+ 140us) write_op.cc:270] Start()
0503 02:12:43.784141 (+ 28us) write_op.cc:276] Timestamp: P: 1714702354689905 usec, L: 0
0503 02:12:43.784335 (+ 194us) write_op.cc:270] Start()
0503 02:12:43.784359 (+ 24us) write_op.cc:276] Timestamp: P: 1714702354690567 usec, L: 0
0503 02:12:43.784499 (+ 140us) write_op.cc:270] Start()
0503 02:12:43.784521 (+ 22us) write_op.cc:276] Timestamp: P: 1714702354691108 usec, L: 0
0503 02:12:43.784683 (+ 162us) write_op.cc:270] Start()
0503 02:12:43.784705 (+ 22us) write_op.cc:276] Timestamp: P: 1714702354698633 usec, L: 0
0503 02:12:43.809142 (+ 24437us) write_op.cc:270] Start()
0503 02:12:43.809189 (+ 47us) write_op.cc:276] Timestamp: P: 1714702354699905 usec, L: 0
0503 02:12:43.809445 (+ 256us) write_op.cc:270] Start()
0503 02:12:43.809469 (+ 24us) write_op.cc:276] Timestamp: P: 1714702354719330 usec, L: 0
0503 02:12:43.809634 (+ 165us) write_op.cc:270] Start()
0503 02:12:43.809653 (+ 19us) write_op.cc:276] Timestamp: P: 1714702354720076 usec, L: 0
0503 02:12:43.809771 (+ 118us) write_op.cc:270] Start()
0503 02:12:43.809790 (+ 19us) write_op.cc:276] Timestamp: P: 1714702354720664 usec, L: 0
0503 02:12:43.809904 (+ 114us) write_op.cc:270] Start()
0503 02:12:43.809920 (+ 16us) write_op.cc:276] Timestamp: P: 1714702354721201 usec, L: 0
0503 02:12:43.810043 (+ 123us) write_op.cc:270] Start()
0503 02:12:43.810062 (+ 19us) write_op.cc:276] Timestamp: P: 1714702354736917 usec, L: 0
0503 02:12:43.810209 (+ 147us) write_op.cc:270] Start()
0503 02:12:43.810229 (+ 20us) write_op.cc:276] Timestamp: P: 1714702354737624 usec, L: 0
0503 02:12:43.843002 (+ 32773us) write_op.cc:270] Start()
0503 02:12:43.843055 (+ 53us) write_op.cc:276] Timestamp: P: 1714702354810153 usec, L: 0
0503 02:12:43.843337 (+ 282us) write_op.cc:270] Start()
0503 02:12:43.843364 (+ 27us) write_op.cc:276] Timestamp: P: 1714702354834157 usec, L: 0
0503 02:12:43.843514 (+ 150us) write_op.cc:270] Start()
0503 02:12:43.843533 (+ 19us) write_op.cc:276] Timestamp: P: 1714702354900488 usec, L: 0
0503 02:12:43.843654 (+ 121us) write_op.cc:270] Start()
0503 02:12:43.843672 (+ 18us) write_op.cc:276] Timestamp: P: 1714702354901309 usec, L: 0
0503 02:12:43.843789 (+ 117us) write_op.cc:270] Start()
0503 02:12:43.843809 (+ 20us) write_op.cc:276] Timestamp: P: 1714702354901970 usec, L: 0
0503 02:12:43.851700 (+ 7891us) write_op.cc:270] Start()
0503 02:12:43.851752 (+ 52us) write_op.cc:276] Timestamp: P: 1714702354902521 usec, L: 0
0503 02:12:43.852013 (+ 261us) write_op.cc:270] Start()
0503 02:12:43.852042 (+ 29us) write_op.cc:276] Timestamp: P: 1714702354903077 usec, L: 0
0503 02:12:43.852205 (+ 163us) write_op.cc:270] Start()
0503 02:12:43.852226 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355011685 usec, L: 0
0503 02:12:43.852357 (+ 131us) write_op.cc:270] Start()
0503 02:12:43.852378 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355012522 usec, L: 0
0503 02:12:43.852508 (+ 130us) write_op.cc:270] Start()
0503 02:12:43.852536 (+ 28us) write_op.cc:276] Timestamp: P: 1714702355032195 usec, L: 0
0503 02:12:43.852676 (+ 140us) write_op.cc:270] Start()
0503 02:12:43.852700 (+ 24us) write_op.cc:276] Timestamp: P: 1714702355033107 usec, L: 0
0503 02:12:43.852846 (+ 146us) write_op.cc:270] Start()
0503 02:12:43.852871 (+ 25us) write_op.cc:276] Timestamp: P: 1714702355079126 usec, L: 0
0503 02:12:43.853075 (+ 204us) write_op.cc:270] Start()
0503 02:12:43.853099 (+ 24us) write_op.cc:276] Timestamp: P: 1714702355080000 usec, L: 0
0503 02:12:43.853267 (+ 168us) write_op.cc:270] Start()
0503 02:12:43.853288 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355080693 usec, L: 0
0503 02:12:43.853450 (+ 162us) write_op.cc:270] Start()
0503 02:12:43.853469 (+ 19us) write_op.cc:276] Timestamp: P: 1714702355081239 usec, L: 0
0503 02:12:43.853631 (+ 162us) write_op.cc:270] Start()
0503 02:12:43.853651 (+ 20us) write_op.cc:276] Timestamp: P: 1714702355082076 usec, L: 0
0503 02:12:43.853814 (+ 163us) write_op.cc:270] Start()
0503 02:12:43.853841 (+ 27us) write_op.cc:276] Timestamp: P: 1714702355130653 usec, L: 0
0503 02:12:43.854015 (+ 174us) write_op.cc:270] Start()
0503 02:12:43.854042 (+ 27us) write_op.cc:276] Timestamp: P: 1714702355137871 usec, L: 0
0503 02:12:43.854182 (+ 140us) write_op.cc:270] Start()
0503 02:12:43.854201 (+ 19us) write_op.cc:276] Timestamp: P: 1714702355138659 usec, L: 0
0503 02:12:43.854329 (+ 128us) write_op.cc:270] Start()
0503 02:12:43.854384 (+ 55us) write_op.cc:276] Timestamp: P: 1714702355142286 usec, L: 0
0503 02:12:43.854502 (+ 118us) write_op.cc:270] Start()
0503 02:12:43.854520 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355143294 usec, L: 0
0503 02:12:43.854645 (+ 125us) write_op.cc:270] Start()
0503 02:12:43.854668 (+ 23us) write_op.cc:276] Timestamp: P: 1714702355144196 usec, L: 0
0503 02:12:43.854801 (+ 133us) write_op.cc:270] Start()
0503 02:12:43.854822 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355145286 usec, L: 0
0503 02:12:43.854998 (+ 176us) write_op.cc:270] Start()
0503 02:12:43.855017 (+ 19us) write_op.cc:276] Timestamp: P: 1714702355146135 usec, L: 0
0503 02:12:43.855129 (+ 112us) write_op.cc:270] Start()
0503 02:12:43.855160 (+ 31us) write_op.cc:276] Timestamp: P: 1714702355147119 usec, L: 0
0503 02:12:43.855297 (+ 137us) write_op.cc:270] Start()
0503 02:12:43.855315 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355148117 usec, L: 0
0503 02:12:43.855432 (+ 117us) write_op.cc:270] Start()
0503 02:12:43.855451 (+ 19us) write_op.cc:276] Timestamp: P: 1714702355191020 usec, L: 0
0503 02:12:43.855582 (+ 131us) write_op.cc:270] Start()
0503 02:12:43.855603 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355217101 usec, L: 0
0503 02:12:43.855731 (+ 128us) write_op.cc:270] Start()
0503 02:12:43.855750 (+ 19us) write_op.cc:276] Timestamp: P: 1714702355217928 usec, L: 0
0503 02:12:43.855921 (+ 171us) write_op.cc:270] Start()
0503 02:12:43.855943 (+ 22us) write_op.cc:276] Timestamp: P: 1714702355244039 usec, L: 0
0503 02:12:43.879413 (+ 23470us) write_op.cc:270] Start()
0503 02:12:43.879479 (+ 66us) write_op.cc:276] Timestamp: P: 1714702355245041 usec, L: 0
0503 02:12:43.879801 (+ 322us) write_op.cc:270] Start()
0503 02:12:43.879830 (+ 29us) write_op.cc:276] Timestamp: P: 1714702355299689 usec, L: 0
0503 02:12:43.879995 (+ 165us) write_op.cc:270] Start()
0503 02:12:43.880024 (+ 29us) write_op.cc:276] Timestamp: P: 1714702355309100 usec, L: 0
0503 02:12:43.880318 (+ 294us) write_op.cc:270] Start()
0503 02:12:43.880353 (+ 35us) write_op.cc:276] Timestamp: P: 1714702355312173 usec, L: 0
0503 02:12:43.880535 (+ 182us) write_op.cc:270] Start()
0503 02:12:43.912519 (+ 31984us) write_op.cc:276] Timestamp: P: 1714702355312806 usec, L: 0
0503 02:12:43.912816 (+ 297us) write_op.cc:270] Start()
0503 02:12:43.912840 (+ 24us) write_op.cc:276] Timestamp: P: 1714702355504944 usec, L: 0
0503 02:12:43.913003 (+ 163us) write_op.cc:270] Start()
0503 02:12:43.913024 (+ 21us) write_op.cc:276] Timestamp: P: 1714702355505644 usec, L: 0
0503 02:12:43.913159 (+ 135us) write_op.cc:270] Start()
0503 02:12:43.913177 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355506214 usec, L: 0
0503 02:12:43.913311 (+ 134us) write_op.cc:270] Start()
0503 02:12:43.913328 (+ 17us) write_op.cc:276] Timestamp: P: 1714702355509564 usec, L: 0
0503 02:12:43.913450 (+ 122us) write_op.cc:270] Start()
0503 02:12:43.913468 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355533587 usec, L: 0
0503 02:12:43.913574 (+ 106us) write_op.cc:270] Start()
0503 02:12:43.913590 (+ 16us) write_op.cc:276] Timestamp: P: 1714702355536042 usec, L: 0
0503 02:12:43.913742 (+ 152us) write_op.cc:270] Start()
0503 02:12:43.913762 (+ 20us) write_op.cc:276] Timestamp: P: 1714702355536818 usec, L: 0
0503 02:12:43.913884 (+ 122us) write_op.cc:270] Start()
0503 02:12:43.913907 (+ 23us) write_op.cc:276] Timestamp: P: 1714702355539190 usec, L: 0
0503 02:12:43.914034 (+ 127us) write_op.cc:270] Start()
0503 02:12:43.914052 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355562402 usec, L: 0
0503 02:12:43.914210 (+ 158us) write_op.cc:270] Start()
0503 02:12:43.914228 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355563071 usec, L: 0
0503 02:12:43.914334 (+ 106us) write_op.cc:270] Start()
0503 02:12:43.914350 (+ 16us) write_op.cc:276] Timestamp: P: 1714702355563662 usec, L: 0
0503 02:12:43.914445 (+ 95us) write_op.cc:270] Start()
0503 02:12:43.914460 (+ 15us) write_op.cc:276] Timestamp: P: 1714702355564165 usec, L: 0
0503 02:12:43.914575 (+ 115us) write_op.cc:270] Start()
0503 02:12:43.914593 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355667608 usec, L: 0
0503 02:12:43.914690 (+ 97us) write_op.cc:270] Start()
0503 02:12:43.914712 (+ 22us) write_op.cc:276] Timestamp: P: 1714702355706509 usec, L: 0
0503 02:12:43.914864 (+ 152us) write_op.cc:270] Start()
0503 02:12:43.914886 (+ 22us) write_op.cc:276] Timestamp: P: 1714702355708865 usec, L: 0
0503 02:12:43.915013 (+ 127us) write_op.cc:270] Start()
0503 02:12:43.915035 (+ 22us) write_op.cc:276] Timestamp: P: 1714702355714019 usec, L: 0
0503 02:12:43.915188 (+ 153us) write_op.cc:270] Start()
0503 02:12:43.915213 (+ 25us) write_op.cc:276] Timestamp: P: 1714702355714643 usec, L: 0
0503 02:12:43.915350 (+ 137us) write_op.cc:270] Start()
0503 02:12:43.915368 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355737465 usec, L: 0
0503 02:12:43.915476 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.915494 (+ 18us) write_op.cc:276] Timestamp: P: 1714702355742871 usec, L: 0
0503 02:12:43.915602 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.915619 (+ 17us) write_op.cc:276] Timestamp: P: 1714702355761189 usec, L: 0
0503 02:12:43.915742 (+ 123us) write_op.cc:270] Start()
0503 02:12:43.915842 (+ 100us) write_op.cc:276] Timestamp: P: 1714702355762261 usec, L: 0
0503 02:12:43.915986 (+ 144us) write_op.cc:270] Start()
0503 02:12:43.916003 (+ 17us) write_op.cc:276] Timestamp: P: 1714702355769902 usec, L: 0
0503 02:12:43.916119 (+ 116us) write_op.cc:270] Start()
0503 02:12:43.916145 (+ 26us) write_op.cc:276] Timestamp: P: 1714702355811119 usec, L: 0
0503 02:12:43.916260 (+ 115us) write_op.cc:270] Start()
0503 02:12:43.916275 (+ 15us) write_op.cc:276] Timestamp: P: 1714702355989741 usec, L: 0
0503 02:12:43.916393 (+ 118us) write_op.cc:270] Start()
0503 02:12:43.916412 (+ 19us) write_op.cc:276] Timestamp: P: 1714702356020486 usec, L: 0
0503 02:12:43.916545 (+ 133us) write_op.cc:270] Start()
0503 02:12:43.916564 (+ 19us) write_op.cc:276] Timestamp: P: 1714702356021388 usec, L: 0
0503 02:12:43.916672 (+ 108us) write_op.cc:270] Start()
0503 02:12:43.916687 (+ 15us) write_op.cc:276] Timestamp: P: 1714702356022168 usec, L: 0
0503 02:12:43.916804 (+ 117us) write_op.cc:270] Start()
0503 02:12:43.916820 (+ 16us) write_op.cc:276] Timestamp: P: 1714702356022938 usec, L: 0
0503 02:12:43.945708 (+ 28888us) write_op.cc:270] Start()
0503 02:12:43.945774 (+ 66us) write_op.cc:276] Timestamp: P: 1714702356026086 usec, L: 0
0503 02:12:43.946819 (+ 1045us) write_op.cc:270] Start()
0503 02:12:43.946853 (+ 34us) write_op.cc:276] Timestamp: P: 1714702356026764 usec, L: 0
0503 02:12:43.947027 (+ 174us) write_op.cc:270] Start()
0503 02:12:43.947051 (+ 24us) write_op.cc:276] Timestamp: P: 1714702356028135 usec, L: 0
0503 02:12:43.947250 (+ 199us) write_op.cc:270] Start()
0503 02:12:43.947273 (+ 23us) write_op.cc:276] Timestamp: P: 1714702356029189 usec, L: 0
0503 02:12:43.947409 (+ 136us) write_op.cc:270] Start()
0503 02:12:43.947433 (+ 24us) write_op.cc:276] Timestamp: P: 1714702356824444 usec, L: 0
0503 02:12:43.947574 (+ 141us) write_op.cc:270] Start()
0503 02:12:43.947591 (+ 17us) write_op.cc:276] Timestamp: P: 1714702356900663 usec, L: 0
0503 02:12:43.947740 (+ 149us) write_op.cc:270] Start()
0503 02:12:43.947766 (+ 26us) write_op.cc:276] Timestamp: P: 1714702356901806 usec, L: 0
0503 02:12:43.947919 (+ 153us) write_op.cc:270] Start()
0503 02:12:43.947942 (+ 23us) write_op.cc:276] Timestamp: P: 1714702356944019 usec, L: 0
0503 02:12:43.948109 (+ 167us) write_op.cc:270] Start()
0503 02:12:43.948143 (+ 34us) write_op.cc:276] Timestamp: P: 1714702356947380 usec, L: 0
0503 02:12:43.948310 (+ 167us) write_op.cc:270] Start()
0503 02:12:43.948335 (+ 25us) write_op.cc:276] Timestamp: P: 1714702356954541 usec, L: 0
0503 02:12:43.948477 (+ 142us) write_op.cc:270] Start()
0503 02:12:43.948499 (+ 22us) write_op.cc:276] Timestamp: P: 1714702356976991 usec, L: 0
0503 02:12:43.970047 (+ 21548us) write_op.cc:270] Start()
0503 02:12:43.970113 (+ 66us) write_op.cc:276] Timestamp: P: 1714702357020611 usec, L: 0
0503 02:12:43.970402 (+ 289us) write_op.cc:270] Start()
0503 02:12:43.970428 (+ 26us) write_op.cc:276] Timestamp: P: 1714702357057142 usec, L: 0
0503 02:12:43.970578 (+ 150us) write_op.cc:270] Start()
0503 02:12:43.970603 (+ 25us) write_op.cc:276] Timestamp: P: 1714702357057869 usec, L: 0
0503 02:12:44.000078 (+ 29475us) log.cc:842] Serialized 118731 byte log entry
0503 02:12:44.030311 (+ 30233us) raft_consensus.cc:1631] Marking committed up to 872
0503 02:12:44.234102 (+203791us) raft_consensus.cc:1683] Filling consensus response to leader.
0503 02:12:44.237426 (+ 3324us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x616000040654. stack: 00007f6afea47756 00007f6afea47510 00007f6afda7a48d 0000000000955804 00000000009557cd 00007f6b0d58715d 00007f6b0d587253 00007f6b0a2b8d41 00007f6b1208d5f5 00007f6b120a0755 00007f6b120a0501 00007f6b0b5bfdbd 00007f6b0a2c927e 00007f6b0a2c8fcd 0000000000a60bcc 00007f6afea8db74
0503 02:12:44.372989 (+135563us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0503 02:12:44.373032 (+ 43us) raft_consensus.cc:1671] finished
0503 02:12:44.373038 (+ 6us) raft_consensus.cc:1677] UpdateReplicas() finished
0503 02:12:44.373286 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:43.348282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:12:43.348518 (+ 236us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:43.348525 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:43.348536 (+ 11us) tablet.cc:660] Decoding operations
0503 02
I20240503 02:12:45.689873 6841 leader_election.cc:288] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.692412 5103 tablet_bootstrap.cc:654] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.701514 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.701781 4807 raft_consensus.cc:2454] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.701840 5105 tablet_bootstrap.cc:492] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:45.702718 4608 leader_election.cc:302] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:45.703076 6829 raft_consensus.cc:2781] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.703198 6829 raft_consensus.cc:482] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.703274 6829 raft_consensus.cc:3037] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.706526 6829 raft_consensus.cc:504] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.707487 6829 leader_election.cc:288] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.707898 5159 ts_tablet_manager.cc:1888] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:45.708048 5159 log.cc:1196] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/9ed525a61c734ee2b23d5d4f319a7d67
I20240503 02:12:45.708446 5159 ts_tablet_manager.cc:1909] T 9ed525a61c734ee2b23d5d4f319a7d67 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:45.710083 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.710912 4807 raft_consensus.cc:3037] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.714486 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 907) took 3614 ms. Trace:
I20240503 02:12:45.714634 5159 rpcz_store.cc:276] 0503 02:12:42.100372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.100507 (+ 135us) service_pool.cc:225] Handling call
0503 02:12:45.400580 (+3300073us) ts_tablet_manager.cc:1120] Deleting tablet 9ed525a61c734ee2b23d5d4f319a7d67
0503 02:12:45.400591 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.576518 (+175927us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.677861 (+101343us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.714450 (+ 36589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3299953}
I20240503 02:12:45.723623 5159 tablet_replica.cc:330] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:45.723856 5159 raft_consensus.cc:2229] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.723999 5159 raft_consensus.cc:2258] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.724560 5159 ts_tablet_manager.cc:1875] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.727311 6861 sys_catalog.cc:455] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "df63329dc2b149d9b08cea341b540d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } } }
I20240503 02:12:45.693172 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:45.708746 4577 tablet.cc:2369] T 1975c790bfa4449baa6d7b81509c5003 P f4f9f079230a468f90e5e3a58f254b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:45.709936 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7c0c5fc614c16985008b8fba54bab" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:45.728935 4608 leader_election.cc:341] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.729212 4608 leader_election.cc:341] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.730542 6861 sys_catalog.cc:458] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:45.734602 6699 raft_consensus.cc:482] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.734782 6699 raft_consensus.cc:504] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.735929 6699 leader_election.cc:288] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.736773 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.736820 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.737031 4810 raft_consensus.cc:2454] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.737584 4608 leader_election.cc:302] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:45.737941 4608 leader_election.cc:341] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.739006 5105 tablet_bootstrap.cc:654] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.739652 6699 raft_consensus.cc:2781] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.740245 6699 raft_consensus.cc:482] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.740372 6699 raft_consensus.cc:3037] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.742774 4807 raft_consensus.cc:2454] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.743301 4608 leader_election.cc:302] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.743640 6830 raft_consensus.cc:2781] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.743746 6830 raft_consensus.cc:686] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.744004 6830 consensus_queue.cc:237] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.744529 5103 tablet_bootstrap.cc:492] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.744688 5103 ts_tablet_manager.cc:1367] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.001s
I20240503 02:12:45.750850 4812 raft_consensus.cc:2454] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.751682 4608 leader_election.cc:302] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.752023 6829 raft_consensus.cc:2781] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.751965 6699 raft_consensus.cc:504] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.752236 6829 raft_consensus.cc:686] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.752630 6829 consensus_queue.cc:237] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.753209 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:45.753926 4608 leader_election.cc:341] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.745635 5103 raft_consensus.cc:348] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.758812 5103 raft_consensus.cc:374] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.758949 5103 raft_consensus.cc:729] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.759232 5103 consensus_queue.cc:260] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.764139 5103 ts_tablet_manager.cc:1398] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:12:45.765491 5159 ts_tablet_manager.cc:1888] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:45.767027 6619 consensus_queue.cc:1035] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.767374 5103 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.769645 5103 tablet_bootstrap.cc:654] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.770082 6699 leader_election.cc:288] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:45.785596 4462 consensus_peers.cc:485] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 035216c9475d4b59aa1ac81b285b4d8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:45.787968 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 9ed525a61c734ee2b23d5d4f319a7d67 (table default.Table_9ffe0b87ba144949b561f827d3d06611 [id=5a08204f484f414c9749befba8b8182b]) successfully deleted
I20240503 02:12:45.798982 6829 raft_consensus.cc:482] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.799356 6829 raft_consensus.cc:504] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.800748 6829 leader_election.cc:288] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.800858 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1436dd3df7284f9bb379236eb9d76836" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.801438 5114 ts_tablet_manager.cc:1909] T 2f862891b43a4845b723bfdb9a7da591 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:45.802186 4608 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.802484 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 900) took 2549 ms. Trace:
I20240503 02:12:45.802649 5114 rpcz_store.cc:276] 0503 02:12:43.253319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.262756 (+ 9437us) service_pool.cc:225] Handling call
0503 02:12:45.461426 (+2198670us) ts_tablet_manager.cc:1120] Deleting tablet 2f862891b43a4845b723bfdb9a7da591
0503 02:12:45.461438 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.530434 (+ 68996us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.657657 (+127223us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.802447 (+144790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2198311}
W20240503 02:12:45.803184 4608 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 1436dd3df7284f9bb379236eb9d76836
I20240503 02:12:45.803309 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 902) took 2420 ms. Trace:
I20240503 02:12:45.803319 4608 leader_election.cc:302] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.803428 5114 rpcz_store.cc:276] 0503 02:12:43.382553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.404039 (+ 21486us) service_pool.cc:225] Handling call
0503 02:12:45.803118 (+2399079us) ts_tablet_manager.cc:1120] Deleting tablet cbaa94834cdc44eabe71242a698e3639
0503 02:12:45.803128 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.803262 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398806}
I20240503 02:12:45.803805 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 903) took 2421 ms. Trace:
I20240503 02:12:45.804016 6829 raft_consensus.cc:2726] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:45.765892 4861 tablet.cc:2369] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:45.808023 5159 log.cc:1196] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/043ba14eaa5b46319b0b88457d2a5f45
I20240503 02:12:45.770618 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfe6b2b9cab4517ae7cabec4796c9ce" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.809093 4805 raft_consensus.cc:3037] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.784751 6862 tablet.cc:1722] T 289300b955dd41c09d2fc348224bb9f4 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:45.809562 5106 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.809731 5106 ts_tablet_manager.cc:1367] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.003s
I20240503 02:12:45.809885 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 2f862891b43a4845b723bfdb9a7da591 (table default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]) successfully deleted
I20240503 02:12:45.816817 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 972cd318ebed41c0ab1b8705f52bdaa4
I20240503 02:12:45.817396 4269 catalog_manager.cc:6554] ReplaceTablet: tablet 972cd318ebed41c0ab1b8705f52bdaa4 deleted and replaced by tablet 07cca97600374070b117e3ef3869042d
I20240503 02:12:45.817757 4269 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c5703a4668b04accbd9b3f93427b31ab" } modify_external_catalogs: true
I20240503 02:12:45.817997 4269 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_c5703a4668b04accbd9b3f93427b31ab" } modify_external_catalogs: true
I20240503 02:12:45.818122 5106 raft_consensus.cc:348] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.818439 5106 raft_consensus.cc:374] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.818499 5106 raft_consensus.cc:729] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.818774 5106 consensus_queue.cc:260] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.818908 4267 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_eee9e761694041d69f22b08ae4363113" } new_table_name: "default.Table_5c564cb4d78e48e4b0bb948f1da64b71" modify_external_catalogs: true
I20240503 02:12:45.819753 5106 ts_tablet_manager.cc:1398] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20240503 02:12:45.821213 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0e7985363654a298c91a0fe701cc3f7 consensus state has no leader
I20240503 02:12:45.821646 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_5d2a8af7b53948258ce2c6f305f318cb" } new_table_name: "default.Table_1f82b7031a824857bfc325fc0c6b1292" modify_external_catalogs: true
I20240503 02:12:45.823681 4275 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6" } modify_external_catalogs: true
I20240503 02:12:45.823853 4275 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_a8a51ba8c1c04eb0a767a80997c7f6f6" } modify_external_catalogs: true
I20240503 02:12:45.824949 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2c2f293e071545c7ae06e32986fbf729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.825507 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c2f293e071545c7ae06e32986fbf729 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.827049 4272 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_57b96feb46144b91851ba8aa2ba702cf" } new_table_name: "default.Table_55b96d43222246e18a8845a6cbb26c56" modify_external_catalogs: true
I20240503 02:12:45.833487 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_3f411ccab7cf48818684560067793bfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.834270 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f411ccab7cf48818684560067793bfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.834990 6699 raft_consensus.cc:482] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:45.835178 6699 raft_consensus.cc:504] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:45.836061 6699 leader_election.cc:288] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:45.837473 4608 leader_election.cc:341] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: 78e9faaa6a584ec38b9da9d2670e47f3
I20240503 02:12:45.866250 5159 ts_tablet_manager.cc:1909] T 043ba14eaa5b46319b0b88457d2a5f45 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:45.871213 5115 ts_tablet_manager.cc:1888] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:45.871448 5115 log.cc:1196] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/59855ab3e42b4e019b769007ccbbc360
I20240503 02:12:45.871932 5115 ts_tablet_manager.cc:1909] T 59855ab3e42b4e019b769007ccbbc360 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:45.872979 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 with delete_type TABLET_DATA_DELETED (replaced by tablet 07cca97600374070b117e3ef3869042d) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:45.873214 4805 raft_consensus.cc:2454] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.874894 4608 leader_election.cc:302] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.875605 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 with delete_type TABLET_DATA_DELETED (replaced by tablet 07cca97600374070b117e3ef3869042d) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:45.875800 6841 raft_consensus.cc:2781] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.876302 6841 raft_consensus.cc:686] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.876600 6841 consensus_queue.cc:237] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:45.883980 4608 leader_election.cc:341] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 78e9faaa6a584ec38b9da9d2670e47f3
I20240503 02:12:45.884166 4608 leader_election.cc:302] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.884485 6830 raft_consensus.cc:2726] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:45.886807 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 908) took 3786 ms. Trace:
I20240503 02:12:45.887010 5159 rpcz_store.cc:276] 0503 02:12:42.100463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.100648 (+ 185us) service_pool.cc:225] Handling call
0503 02:12:45.714992 (+3614344us) ts_tablet_manager.cc:1120] Deleting tablet 043ba14eaa5b46319b0b88457d2a5f45
0503 02:12:45.716012 (+ 1020us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.745417 (+ 29405us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.765367 (+ 19950us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.886782 (+121415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3614135}
I20240503 02:12:45.887698 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 915) took 3678 ms. Trace:
I20240503 02:12:45.887856 5159 rpcz_store.cc:276] 0503 02:12:42.209173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.213542 (+ 4369us) service_pool.cc:225] Handling call
0503 02:12:45.887570 (+3674028us) ts_tablet_manager.cc:1120] Deleting tablet 1a082f4c43fd4c188be9ee3966e26edf
0503 02:12:45.887579 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.887664 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3673805}
I20240503 02:12:45.888031 5105 tablet_bootstrap.cc:492] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:45.888191 5105 ts_tablet_manager.cc:1367] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.012s
I20240503 02:12:45.888484 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 916) took 3679 ms. Trace:
I20240503 02:12:45.888605 5159 rpcz_store.cc:276] 0503 02:12:42.209294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.213805 (+ 4511us) service_pool.cc:225] Handling call
0503 02:12:45.888381 (+3674576us) ts_tablet_manager.cc:1120] Deleting tablet d99a7e930a3947b4aae62424f11d7227
0503 02:12:45.888390 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.888467 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3674446}
I20240503 02:12:45.888829 5105 raft_consensus.cc:348] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.889068 5105 raft_consensus.cc:374] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.889142 5105 raft_consensus.cc:729] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.889242 5159 tablet_replica.cc:330] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:45.889359 5105 consensus_queue.cc:260] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.889508 5159 raft_consensus.cc:2229] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.889773 5159 raft_consensus.cc:2258] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.889874 5105 ts_tablet_manager.cc:1398] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:45.890166 5105 tablet_bootstrap.cc:492] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:45.890704 5159 ts_tablet_manager.cc:1875] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.870497 6769 consensus_queue.cc:1035] T d9dfb4150c3c49f4b5d4e31396da4e8d P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.874281 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 972cd318ebed41c0ab1b8705f52bdaa4 with delete_type TABLET_DATA_DELETED (replaced by tablet 07cca97600374070b117e3ef3869042d) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.874310 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f39e8d17b44895bd5c979d82145ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:45.891738 5114 rpcz_store.cc:276] 0503 02:12:43.382711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.404382 (+ 21671us) service_pool.cc:225] Handling call
0503 02:12:45.803670 (+2399288us) ts_tablet_manager.cc:1120] Deleting tablet f8be9e5380b4466baccd16483e940588
0503 02:12:45.803679 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.803767 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399117}
I20240503 02:12:45.892100 5114 tablet_replica.cc:330] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:45.892313 5114 raft_consensus.cc:2229] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.892510 5114 raft_consensus.cc:2258] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.893167 5114 ts_tablet_manager.cc:1875] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.897361 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 772) took 5889 ms. Trace:
I20240503 02:12:45.897485 5106 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.897679 5115 rpcz_store.cc:276] 0503 02:12:40.007439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.021404 (+ 13965us) service_pool.cc:225] Handling call
0503 02:12:45.116491 (+5095087us) ts_tablet_manager.cc:1120] Deleting tablet 59855ab3e42b4e019b769007ccbbc360
0503 02:12:45.116502 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.498323 (+381821us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.664083 (+165760us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.897326 (+233243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5094793}
I20240503 02:12:45.899323 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 785) took 5529 ms. Trace:
I20240503 02:12:45.899735 5115 rpcz_store.cc:276] 0503 02:12:40.370192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.370249 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:45.898814 (+5528565us) ts_tablet_manager.cc:1120] Deleting tablet 13e31219b2134fa080572238ff0be424
0503 02:12:45.898825 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.899215 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5528246}
I20240503 02:12:45.901342 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 786) took 5530 ms. Trace:
I20240503 02:12:45.877343 5103 tablet_bootstrap.cc:492] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.902884 5105 tablet_bootstrap.cc:654] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.903959 5106 tablet_bootstrap.cc:654] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.911264 5103 ts_tablet_manager.cc:1367] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.135s user 0.018s sys 0.015s
I20240503 02:12:45.913949 5105 tablet_bootstrap.cc:492] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:45.914115 5105 ts_tablet_manager.cc:1367] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20240503 02:12:45.914005 5106 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:45.914465 5106 ts_tablet_manager.cc:1367] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20240503 02:12:45.914932 5105 raft_consensus.cc:348] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.915174 5105 raft_consensus.cc:374] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.915239 5105 raft_consensus.cc:729] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.915283 6738 raft_consensus.cc:482] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.915400 6738 raft_consensus.cc:504] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.915474 5105 consensus_queue.cc:260] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.915513 5106 raft_consensus.cc:348] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.915871 5106 raft_consensus.cc:374] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.915937 5106 raft_consensus.cc:729] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.915935 5105 ts_tablet_manager.cc:1398] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:45.915998 6738 leader_election.cc:288] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:45.916215 5105 tablet_bootstrap.cc:492] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:45.916296 5106 consensus_queue.cc:260] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.916801 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa67857b858c4719b498291c52b7aceb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.901731 5115 rpcz_store.cc:276] 0503 02:12:40.371002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.371052 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:45.900798 (+5529746us) ts_tablet_manager.cc:1120] Deleting tablet 73198ec769a9486da3f85e8fcbef8e78
0503 02:12:45.900895 (+ 97us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.901276 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5529496}
W20240503 02:12:45.917416 4746 leader_election.cc:341] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.919162 5105 tablet_bootstrap.cc:654] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.922250 6841 raft_consensus.cc:482] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.922497 6841 raft_consensus.cc:504] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.923689 6841 leader_election.cc:288] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.916720 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa67857b858c4719b498291c52b7aceb" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.923888 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:45.917212 5106 ts_tablet_manager.cc:1398] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240503 02:12:45.924355 4608 leader_election.cc:341] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:45.924794 4746 leader_election.cc:341] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.924945 5106 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.924970 4746 leader_election.cc:302] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.925511 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 787) took 5526 ms. Trace:
I20240503 02:12:45.925663 6814 raft_consensus.cc:2726] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.925595 5115 rpcz_store.cc:276] 0503 02:12:40.398858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.399014 (+ 156us) service_pool.cc:225] Handling call
0503 02:12:45.924242 (+5525228us) ts_tablet_manager.cc:1120] Deleting tablet a8ecddfcfed2436e98bb7b6b38beb03f
0503 02:12:45.924257 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.925488 (+ 1231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5524981}
I20240503 02:12:45.925995 6862 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:45.926571 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 788) took 5499 ms. Trace:
I20240503 02:12:45.926793 5106 tablet_bootstrap.cc:654] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.918684 5103 raft_consensus.cc:348] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.927287 6857 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:45.926666 5115 rpcz_store.cc:276] 0503 02:12:40.427428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.427503 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:45.925855 (+5498352us) ts_tablet_manager.cc:1120] Deleting tablet 3b92999890ee49d78d7c16f5ed1c4958
0503 02:12:45.925864 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.926550 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5498104}
I20240503 02:12:45.928669 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 789) took 5500 ms. Trace:
I20240503 02:12:45.928809 5115 rpcz_store.cc:276] 0503 02:12:40.428546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.428712 (+ 166us) service_pool.cc:225] Handling call
0503 02:12:45.928562 (+5499850us) ts_tablet_manager.cc:1120] Deleting tablet fca2c34e0fc44be8b7ee9252ef557497
0503 02:12:45.928571 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.928653 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5499661}
I20240503 02:12:45.929350 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 790) took 5479 ms. Trace:
I20240503 02:12:45.929474 5115 rpcz_store.cc:276] 0503 02:12:40.449681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.449737 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:45.929258 (+5479521us) ts_tablet_manager.cc:1120] Deleting tablet c919bda1eaaa4e52828aac3ef337f3e5
0503 02:12:45.929264 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.929336 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5479282}
I20240503 02:12:45.929944 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 791) took 5242 ms. Trace:
I20240503 02:12:45.930068 5115 rpcz_store.cc:276] 0503 02:12:40.687936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.688484 (+ 548us) service_pool.cc:225] Handling call
0503 02:12:45.929854 (+5241370us) ts_tablet_manager.cc:1120] Deleting tablet 07225ae9ad3c4e44ac0c5c51421b0115
0503 02:12:45.929862 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.929929 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5241171}
I20240503 02:12:45.930809 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:45.931020 4809 raft_consensus.cc:2454] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:45.931969 4608 leader_election.cc:302] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:45.932689 6830 raft_consensus.cc:2781] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.932762 6830 raft_consensus.cc:482] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.932819 6830 raft_consensus.cc:3037] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.935664 6830 raft_consensus.cc:504] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.936479 6830 leader_election.cc:288] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.937798 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:45.938022 4812 raft_consensus.cc:3037] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.941625 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf67d8c1d3ec4f758cca9745c42c7ee6" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:45.942499 4812 raft_consensus.cc:2454] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:45.943243 4608 leader_election.cc:302] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:45.943554 6841 raft_consensus.cc:2781] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:45.943599 4608 leader_election.cc:341] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.943688 6841 raft_consensus.cc:686] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:45.943972 6841 consensus_queue.cc:237] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.949378 5159 ts_tablet_manager.cc:1888] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:45.952277 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 59855ab3e42b4e019b769007ccbbc360 (table default.Table_8ed808d35bcc41bab738b11c2deca02c [id=ea03e4da35f34245942edec5efba152a]) successfully deleted
I20240503 02:12:45.954188 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 792) took 5264 ms. Trace:
W20240503 02:12:45.955032 4267 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 consensus state has no leader
I20240503 02:12:45.954311 5115 rpcz_store.cc:276] 0503 02:12:40.690136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.690190 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:45.943306 (+5253116us) ts_tablet_manager.cc:1120] Deleting tablet 0ab2b01fb13c4aaa931b3a633ab5d847
0503 02:12:45.943319 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.954143 (+ 10824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5252943}
I20240503 02:12:45.955801 6619 raft_consensus.cc:482] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.927310 5103 raft_consensus.cc:374] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.955940 5103 raft_consensus.cc:729] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.955983 6619 raft_consensus.cc:504] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.956251 5103 consensus_queue.cc:260] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:45.956357 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 452d494e14c440e6805055ba333b4e7f consensus state has no leader
I20240503 02:12:45.956714 6619 leader_election.cc:288] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.956847 5103 ts_tablet_manager.cc:1398] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.039s user 0.001s sys 0.000s
I20240503 02:12:45.957391 5103 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
W20240503 02:12:45.958266 4462 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:45.959124 5103 tablet_bootstrap.cc:654] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:45.960521 4272 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e489b424b2d546e2aa993163f13dcbd9 consensus state has no leader
I20240503 02:12:45.960961 4272 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_64cb0d91e91245c3a93537f8b7d4f778" } modify_external_catalogs: true
I20240503 02:12:45.961166 4272 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_64cb0d91e91245c3a93537f8b7d4f778" } modify_external_catalogs: true
I20240503 02:12:45.961856 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f39e8d17b44895bd5c979d82145ad9" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.962127 4667 raft_consensus.cc:2454] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:45.962070 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6d1f952d30ef4ea4ba59b6b7ebf7548d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.962646 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d1f952d30ef4ea4ba59b6b7ebf7548d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.962747 4462 leader_election.cc:302] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:45.963279 6619 raft_consensus.cc:2781] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.963408 6619 raft_consensus.cc:482] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.963519 6619 raft_consensus.cc:3037] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.966226 4270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_0f2571bf88ac4a23b6b806c2ae05671b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:45.966840 4270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f2571bf88ac4a23b6b806c2ae05671b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:45.967396 6619 raft_consensus.cc:504] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.967744 5103 tablet_bootstrap.cc:492] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:45.967887 5103 ts_tablet_manager.cc:1367] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:12:45.968168 5114 ts_tablet_manager.cc:1888] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:45.968286 6619 leader_election.cc:288] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.968760 5103 raft_consensus.cc:348] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.968981 5103 raft_consensus.cc:374] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.969601 6619 raft_consensus.cc:482] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.969729 6619 raft_consensus.cc:504] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.970010 5106 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:45.970168 4462 leader_election.cc:341] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:45.970191 5106 ts_tablet_manager.cc:1367] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.001s
I20240503 02:12:45.970312 6619 leader_election.cc:288] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.969061 5103 raft_consensus.cc:729] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.970461 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:45.970702 4266 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_76d4f904da56406595b0e3da677e7cc8 [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 6 got 5
I20240503 02:12:45.970628 5103 consensus_queue.cc:260] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.970707 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:45.970935 5106 raft_consensus.cc:348] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.971204 5103 ts_tablet_manager.cc:1398] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:45.971205 5106 raft_consensus.cc:374] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.971280 5106 raft_consensus.cc:729] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.971503 5103 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:45.971544 5106 consensus_queue.cc:260] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:45.972052 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f39e8d17b44895bd5c979d82145ad9" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:45.972231 4667 raft_consensus.cc:3037] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.973001 5106 ts_tablet_manager.cc:1398] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:45.973148 5103 tablet_bootstrap.cc:654] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.975657 4667 raft_consensus.cc:2454] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:45.975656 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_bea552b4d4954b9b80d91b223a4349ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:45.976316 4462 leader_election.cc:302] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:45.976751 6619 raft_consensus.cc:2781] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.976989 6619 raft_consensus.cc:686] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:45.977283 6619 consensus_queue.cc:237] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:45.980787 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bea552b4d4954b9b80d91b223a4349ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:12:45.983455 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 13e31219b2134fa080572238ff0be424 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13e31219b2134fa080572238ff0be424 already in progress: creating tablet
W20240503 02:12:45.983768 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 73198ec769a9486da3f85e8fcbef8e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73198ec769a9486da3f85e8fcbef8e78 already in progress: creating tablet
W20240503 02:12:45.984050 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a8ecddfcfed2436e98bb7b6b38beb03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ecddfcfed2436e98bb7b6b38beb03f already in progress: creating tablet
W20240503 02:12:45.984293 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3b92999890ee49d78d7c16f5ed1c4958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b92999890ee49d78d7c16f5ed1c4958 already in progress: creating tablet
W20240503 02:12:45.984550 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet fca2c34e0fc44be8b7ee9252ef557497 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca2c34e0fc44be8b7ee9252ef557497 already in progress: creating tablet
W20240503 02:12:45.984800 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet c919bda1eaaa4e52828aac3ef337f3e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c919bda1eaaa4e52828aac3ef337f3e5 already in progress: creating tablet
I20240503 02:12:45.985096 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 793) took 5287 ms. Trace:
W20240503 02:12:45.985209 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 07225ae9ad3c4e44ac0c5c51421b0115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07225ae9ad3c4e44ac0c5c51421b0115 already in progress: creating tablet
I20240503 02:12:45.985217 5115 rpcz_store.cc:276] 0503 02:12:40.697715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.697772 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:45.956306 (+5258534us) ts_tablet_manager.cc:1120] Deleting tablet 676d5f52102d496081df3964722b286f
0503 02:12:45.984838 (+ 28532us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.985054 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5258315}
W20240503 02:12:45.985466 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0ab2b01fb13c4aaa931b3a633ab5d847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ab2b01fb13c4aaa931b3a633ab5d847 already in progress: creating tablet
I20240503 02:12:45.985652 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 794) took 5284 ms. Trace:
W20240503 02:12:45.985735 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet f8be9e5380b4466baccd16483e940588 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8be9e5380b4466baccd16483e940588 already in progress: creating tablet
I20240503 02:12:45.985718 5115 rpcz_store.cc:276] 0503 02:12:40.701449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.701498 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:45.985568 (+5284070us) ts_tablet_manager.cc:1120] Deleting tablet 22776c81d1a24d368abb68274cf32a0b
0503 02:12:45.985578 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.985636 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5283885}
I20240503 02:12:45.985920 5115 tablet_replica.cc:330] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:45.986090 5115 raft_consensus.cc:2229] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.986105 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 043ba14eaa5b46319b0b88457d2a5f45 (table default.Table_155fde3f789341069f58c427a2c25eec [id=73b33f37dc3e4630a5935314a7597445]) successfully deleted
I20240503 02:12:45.986243 5115 raft_consensus.cc:2258] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:45.986335 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1a082f4c43fd4c188be9ee3966e26edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a082f4c43fd4c188be9ee3966e26edf already in progress: creating tablet
W20240503 02:12:45.986550 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet d99a7e930a3947b4aae62424f11d7227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d99a7e930a3947b4aae62424f11d7227 already in progress: creating tablet
I20240503 02:12:45.986799 5115 ts_tablet_manager.cc:1875] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.989742 5106 tablet_bootstrap.cc:492] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:45.991684 5106 tablet_bootstrap.cc:654] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.996433 6841 raft_consensus.cc:482] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:45.996623 6841 raft_consensus.cc:504] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } }
I20240503 02:12:45.997562 6841 leader_election.cc:288] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:45.998839 5115 ts_tablet_manager.cc:1888] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:45.998951 5115 log.cc:1196] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/78e9faaa6a584ec38b9da9d2670e47f3
W20240503 02:12:45.999104 4608 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d39f042afb954daa923502ef12377bd5
I20240503 02:12:45.999375 5115 ts_tablet_manager.cc:1909] T 78e9faaa6a584ec38b9da9d2670e47f3 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:45.999788 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 795) took 5294 ms. Trace:
I20240503 02:12:45.999876 5115 rpcz_store.cc:276] 0503 02:12:40.705463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.705516 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:45.985878 (+5280362us) ts_tablet_manager.cc:1120] Deleting tablet 78e9faaa6a584ec38b9da9d2670e47f3
0503 02:12:45.985885 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.994645 (+ 8760us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.998722 (+ 4077us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.999767 (+ 1045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5280164}
I20240503 02:12:46.000332 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 799) took 5194 ms. Trace:
W20240503 02:12:46.000371 4608 leader_election.cc:341] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: d39f042afb954daa923502ef12377bd5
I20240503 02:12:46.000458 4608 leader_election.cc:302] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.000999 6841 raft_consensus.cc:2726] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:12:46.001727 4261 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 (table default.Table_5c564cb4d78e48e4b0bb948f1da64b71 [id=a551765407ab434fa2f1e8161e9df406], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1ba685e1c1324a3e8ad9a8eb8847ab87 consensus state has no leader
I20240503 02:12:46.001775 6831 tablet.cc:1722] T bd0d62ff4ac2460a9620a8e16e5d6548 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 02:12:46.002494 6769 raft_consensus.cc:482] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.002818 6769 raft_consensus.cc:504] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.003571 6769 leader_election.cc:288] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.000425 5115 rpcz_store.cc:276] 0503 02:12:40.806196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.806256 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:46.000184 (+5193928us) ts_tablet_manager.cc:1120] Deleting tablet b8287b45fa0b492ca82c22164f3a7837
0503 02:12:46.000194 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.000317 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5193736}
I20240503 02:12:46.004398 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 807) took 5111 ms. Trace:
I20240503 02:12:46.004498 5115 rpcz_store.cc:276] 0503 02:12:40.892832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:40.893292 (+ 460us) service_pool.cc:225] Handling call
0503 02:12:46.004265 (+5110973us) ts_tablet_manager.cc:1120] Deleting tablet a48d5eac44264899a7a372b9f85fb8f5
0503 02:12:46.004275 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.004382 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5110812}
W20240503 02:12:46.004700 4462 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 27c13f257eae44689af07cc31ea254c4
I20240503 02:12:46.006341 5106 tablet_bootstrap.cc:492] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.006546 5106 ts_tablet_manager.cc:1367] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240503 02:12:46.007354 5106 raft_consensus.cc:348] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.007609 5106 raft_consensus.cc:374] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.007524 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c13f257eae44689af07cc31ea254c4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.007700 5106 raft_consensus.cc:729] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.007798 4667 raft_consensus.cc:2454] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:46.008108 5106 consensus_queue.cc:260] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.008337 4462 leader_election.cc:302] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:46.008831 6769 raft_consensus.cc:2781] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.009043 6769 raft_consensus.cc:482] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.009181 5115 tablet_replica.cc:330] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.009224 6769 raft_consensus.cc:3037] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.009399 5115 raft_consensus.cc:2229] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.009763 5115 raft_consensus.cc:2258] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.011745 5115 ts_tablet_manager.cc:1875] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:45.968304 5114 log.cc:1196] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/88b173d78ba74ede801b12cccbeef6b1
I20240503 02:12:46.018908 5106 ts_tablet_manager.cc:1398] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:12:46.019415 5106 tablet_bootstrap.cc:492] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.019488 5114 ts_tablet_manager.cc:1909] T 88b173d78ba74ede801b12cccbeef6b1 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:46.020174 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 676d5f52102d496081df3964722b286f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676d5f52102d496081df3964722b286f already in progress: creating tablet
I20240503 02:12:46.021546 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 906) took 2448 ms. Trace:
W20240503 02:12:46.022063 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 22776c81d1a24d368abb68274cf32a0b because the tablet was not found. No further retry: Not found: Tablet not found: 22776c81d1a24d368abb68274cf32a0b
I20240503 02:12:46.022553 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 78e9faaa6a584ec38b9da9d2670e47f3 (table default.Table_f07e8b862b79406bab7c9aa1e63fd2f8 [id=e1adcda28d7a459db219dc925bd7e6a3]) successfully deleted
I20240503 02:12:46.026479 5106 tablet_bootstrap.cc:654] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.028373 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet bd0d62ff4ac2460a9620a8e16e5d6548 (table default.Table_76d4f904da56406595b0e3da677e7cc8 [id=4d9a163c0c564955a47a65f799a0018c]). Expected version 6 got 5
I20240503 02:12:46.036187 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet de4e96b4be094101bf74904b2d66b6c7 (table default.Table_0f920eca3cce4751b3898c378a8efd18 [id=166d3e6f76994946a92d66855ec30544]). Expected version 1 got 0
I20240503 02:12:46.038673 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:46.039115 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:46.039753 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 8c27e37ebd0f49cdba7653548acbe1ea (table default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129]). Expected version 1 got 0
I20240503 02:12:46.040220 4268 catalog_manager.cc:5484] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.042482 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:45.955694 5105 tablet_bootstrap.cc:492] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:45.971625 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2efdbfa909c475d95e99cf24206608f" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.045838 5159 log.cc:1196] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/42e80cc31f5c4908b82f2b030255f6af
I20240503 02:12:46.045930 5105 ts_tablet_manager.cc:1367] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.005s
I20240503 02:12:45.982141 6738 raft_consensus.cc:482] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.046147 6738 raft_consensus.cc:504] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.046289 5159 ts_tablet_manager.cc:1909] T 42e80cc31f5c4908b82f2b030255f6af P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:46.046651 5105 raft_consensus.cc:348] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.046779 6738 leader_election.cc:288] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.046834 5105 raft_consensus.cc:374] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.046880 5105 raft_consensus.cc:729] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.047068 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 917) took 3837 ms. Trace:
I20240503 02:12:46.036245 6814 raft_consensus.cc:482] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.047163 5159 rpcz_store.cc:276] 0503 02:12:42.209433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.213961 (+ 4528us) service_pool.cc:225] Handling call
0503 02:12:45.889035 (+3675074us) ts_tablet_manager.cc:1120] Deleting tablet 42e80cc31f5c4908b82f2b030255f6af
0503 02:12:45.889044 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.932696 (+ 43652us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.949230 (+ 16534us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.047047 (+ 97817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3674938}
I20240503 02:12:46.047353 6814 raft_consensus.cc:504] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.047988 6814 leader_election.cc:288] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.047082 5105 consensus_queue.cc:260] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.048622 5105 ts_tablet_manager.cc:1398] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:46.048933 5105 tablet_bootstrap.cc:492] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.049450 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efef931ca554ae18579bbea8d1c04f7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:46.050076 4746 leader_election.cc:341] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.050345 5105 tablet_bootstrap.cc:654] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.051991 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee34be89bda450296b916163bb10c46" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:46.052780 4746 leader_election.cc:341] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.047320 4462 leader_election.cc:302] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:46.047850 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efef931ca554ae18579bbea8d1c04f7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.056402 6811 raft_consensus.cc:2726] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.055002 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee34be89bda450296b916163bb10c46" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:46.057142 4746 leader_election.cc:341] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.057284 4746 leader_election.cc:302] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.057374 6774 catalog_manager.cc:1259] Loaded cluster ID: fd811578fb3d46819c1d78577e001af2
I20240503 02:12:46.057477 6774 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: loading cluster ID for follower catalog manager: success
W20240503 02:12:46.057807 4746 leader_election.cc:341] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.057960 4746 leader_election.cc:302] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.058457 6738 raft_consensus.cc:2726] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.058650 6738 raft_consensus.cc:2726] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:46.023134 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b8287b45fa0b492ca82c22164f3a7837 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8287b45fa0b492ca82c22164f3a7837 already in progress: creating tablet
I20240503 02:12:46.064558 5114 rpcz_store.cc:276] 0503 02:12:43.572616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.572671 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:45.892028 (+2319357us) ts_tablet_manager.cc:1120] Deleting tablet 88b173d78ba74ede801b12cccbeef6b1
0503 02:12:45.892040 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:45.958000 (+ 65960us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:45.968019 (+ 10019us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.021507 (+ 53488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319153}
W20240503 02:12:46.065518 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a48d5eac44264899a7a372b9f85fb8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48d5eac44264899a7a372b9f85fb8f5 already in progress: creating tablet
I20240503 02:12:46.066157 5114 tablet_replica.cc:330] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.048139 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 918) took 3828 ms. Trace:
I20240503 02:12:46.069422 5159 rpcz_store.cc:276] 0503 02:12:42.219418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.219476 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:46.047599 (+3828123us) ts_tablet_manager.cc:1120] Deleting tablet b7d0640cea2a482e9974376ba479163b
0503 02:12:46.047609 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.047725 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827921}
I20240503 02:12:46.058889 5115 ts_tablet_manager.cc:1888] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.069955 5115 log.cc:1196] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3db94fb9d53044d796e2973143d888fb
I20240503 02:12:46.070585 5115 ts_tablet_manager.cc:1909] T 3db94fb9d53044d796e2973143d888fb P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.073252 5114 raft_consensus.cc:2229] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.074435 5114 raft_consensus.cc:2258] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.074748 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cf0d0e6e043bab333c835570b4e0c (DEFAULT_TABLE table=default.Table_5f1814596fb2492f9b62c8d2befb76e9 [id=dd7fd0e3f9aa460f88c1c67fcc126aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.075192 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cf0d0e6e043bab333c835570b4e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.075392 5114 ts_tablet_manager.cc:1875] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.075650 6841 raft_consensus.cc:482] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.075815 6841 raft_consensus.cc:504] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.076500 6841 leader_election.cc:288] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.078307 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cf0d0e6e043bab333c835570b4e0c (DEFAULT_TABLE table=default.Table_5f1814596fb2492f9b62c8d2befb76e9 [id=dd7fd0e3f9aa460f88c1c67fcc126aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.078842 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 394b407592dc4d8885bdaca3dfc6bea4 (DEFAULT_TABLE table=default.Table_2ff3d2b4b7054749b3d29eefd67d6abf [id=c0f8a7222c2b4e34a19de12469b2a159]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.078879 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 394b407592dc4d8885bdaca3dfc6bea4 (DEFAULT_TABLE table=default.Table_2ff3d2b4b7054749b3d29eefd67d6abf [id=c0f8a7222c2b4e34a19de12469b2a159]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.079277 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394b407592dc4d8885bdaca3dfc6bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.081329 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cf0d0e6e043bab333c835570b4e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.081897 4636 tablet_service.cc:1467] Processing CreateTablet for tablet c9068cb6a3b84688b5f53ba3e6ec03d1 (DEFAULT_TABLE table=default.Table_d0a57800fa3a48808a5f01bd6bb4945b [id=0b95c35faafb4d2badcc85f28ca86dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.082257 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9068cb6a3b84688b5f53ba3e6ec03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.082984 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cf0d0e6e043bab333c835570b4e0c (DEFAULT_TABLE table=default.Table_5f1814596fb2492f9b62c8d2befb76e9 [id=dd7fd0e3f9aa460f88c1c67fcc126aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.084085 6841 raft_consensus.cc:482] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.084231 6841 raft_consensus.cc:504] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.085033 6841 leader_election.cc:288] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.084995 4507 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84c48a64814f65ab774458ba44ccc3 (DEFAULT_TABLE table=default.Table_74b07d82f6514260885515c05151bec9 [id=fc76f69997684871b5683732cf50db0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.085170 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394b407592dc4d8885bdaca3dfc6bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.085373 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84c48a64814f65ab774458ba44ccc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.085788 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.086395 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:46.086817 4608 leader_election.cc:341] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:46.087152 4608 leader_election.cc:341] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.088025 6811 consensus_queue.cc:1035] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.089286 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84c48a64814f65ab774458ba44ccc3 (DEFAULT_TABLE table=default.Table_74b07d82f6514260885515c05151bec9 [id=fc76f69997684871b5683732cf50db0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.089727 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84c48a64814f65ab774458ba44ccc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.090466 4497 tablet_service.cc:1467] Processing CreateTablet for tablet c9068cb6a3b84688b5f53ba3e6ec03d1 (DEFAULT_TABLE table=default.Table_d0a57800fa3a48808a5f01bd6bb4945b [id=0b95c35faafb4d2badcc85f28ca86dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.090694 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 45703baa79ea4aaa8d9181bc25480b70 (DEFAULT_TABLE table=default.Table_adfe1adf426343a2ace07860d42d9083 [id=f200890326914da7b874573c3babe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.090862 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9068cb6a3b84688b5f53ba3e6ec03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.091055 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45703baa79ea4aaa8d9181bc25480b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.091249 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cf0d0e6e043bab333c835570b4e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.094666 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 814) took 4981 ms. Trace:
I20240503 02:12:46.094779 5115 rpcz_store.cc:276] 0503 02:12:41.113510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.113571 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:46.009120 (+4895549us) ts_tablet_manager.cc:1120] Deleting tablet 3db94fb9d53044d796e2973143d888fb
0503 02:12:46.009132 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.055521 (+ 46389us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.058776 (+ 3255us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.094638 (+ 35862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4895271}
I20240503 02:12:46.095139 5115 tablet_replica.cc:330] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.095347 5115 raft_consensus.cc:2229] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.099974 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.100261 4812 raft_consensus.cc:2454] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.101182 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 62a870f37ae1458e978307e0d3505711 (DEFAULT_TABLE table=default.Table_94b433051bd4452cbb4566bba7d627a3 [id=e0cef1fa41ed4f5392486882b9f13edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.101222 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 394b407592dc4d8885bdaca3dfc6bea4 (DEFAULT_TABLE table=default.Table_2ff3d2b4b7054749b3d29eefd67d6abf [id=c0f8a7222c2b4e34a19de12469b2a159]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.101603 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394b407592dc4d8885bdaca3dfc6bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.101604 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a870f37ae1458e978307e0d3505711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.101892 4776 tablet_service.cc:1467] Processing CreateTablet for tablet c9068cb6a3b84688b5f53ba3e6ec03d1 (DEFAULT_TABLE table=default.Table_d0a57800fa3a48808a5f01bd6bb4945b [id=0b95c35faafb4d2badcc85f28ca86dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.104807 5115 raft_consensus.cc:2258] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.105537 5115 ts_tablet_manager.cc:1875] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.108611 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84c48a64814f65ab774458ba44ccc3 (DEFAULT_TABLE table=default.Table_74b07d82f6514260885515c05151bec9 [id=fc76f69997684871b5683732cf50db0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.108965 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 45703baa79ea4aaa8d9181bc25480b70 (DEFAULT_TABLE table=default.Table_adfe1adf426343a2ace07860d42d9083 [id=f200890326914da7b874573c3babe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.110219 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84c48a64814f65ab774458ba44ccc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.114557 4507 tablet_service.cc:1467] Processing CreateTablet for tablet a673151d0db649a0914141f7623d11ed (DEFAULT_TABLE table=default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.115038 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a673151d0db649a0914141f7623d11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.117723 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9068cb6a3b84688b5f53ba3e6ec03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.117652 4608 leader_election.cc:302] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.118328 6841 raft_consensus.cc:2781] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.118423 6841 raft_consensus.cc:482] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.118479 6841 raft_consensus.cc:3037] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.121101 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45703baa79ea4aaa8d9181bc25480b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.121135 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 62a870f37ae1458e978307e0d3505711 (DEFAULT_TABLE table=default.Table_94b433051bd4452cbb4566bba7d627a3 [id=e0cef1fa41ed4f5392486882b9f13edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.120805 4645 tablet_service.cc:1467] Processing CreateTablet for tablet 45703baa79ea4aaa8d9181bc25480b70 (DEFAULT_TABLE table=default.Table_adfe1adf426343a2ace07860d42d9083 [id=f200890326914da7b874573c3babe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.124998 4639 tablet_service.cc:1467] Processing CreateTablet for tablet a673151d0db649a0914141f7623d11ed (DEFAULT_TABLE table=default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.125092 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.126660 4507 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3ca23bd0149d396950a340632898d (DEFAULT_TABLE table=default.Table_9ed74e0af4724d85a61a10acc9942836 [id=0837945579074879b3c9e79030b099fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.127204 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3ca23bd0149d396950a340632898d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.128787 6830 raft_consensus.cc:482] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.128968 6830 raft_consensus.cc:504] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.129684 6830 leader_election.cc:288] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.129835 4651 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3ca23bd0149d396950a340632898d (DEFAULT_TABLE table=default.Table_9ed74e0af4724d85a61a10acc9942836 [id=0837945579074879b3c9e79030b099fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.132962 4789 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3ca23bd0149d396950a340632898d (DEFAULT_TABLE table=default.Table_9ed74e0af4724d85a61a10acc9942836 [id=0837945579074879b3c9e79030b099fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.133433 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3ca23bd0149d396950a340632898d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.133477 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.133579 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1fc5c43794b0b8cf4d9650fcadef2 (DEFAULT_TABLE table=default.Table_2f9fb44984074b81baaac14bf2486c1c [id=2677b3a7e24a4bf8b897583cf2f94a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.133919 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1fc5c43794b0b8cf4d9650fcadef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.134627 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.134817 4812 raft_consensus.cc:2454] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:46.135591 4608 leader_election.cc:341] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.135726 4608 leader_election.cc:302] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.136142 6830 raft_consensus.cc:2726] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.136852 4608 leader_election.cc:302] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:46.137221 4608 leader_election.cc:341] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.137574 6830 raft_consensus.cc:2781] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.137656 6830 raft_consensus.cc:482] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.137722 6830 raft_consensus.cc:3037] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.139001 4784 tablet_service.cc:1467] Processing CreateTablet for tablet a673151d0db649a0914141f7623d11ed (DEFAULT_TABLE table=default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.139448 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a673151d0db649a0914141f7623d11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.140507 6830 raft_consensus.cc:504] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.141527 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1fc5c43794b0b8cf4d9650fcadef2 (DEFAULT_TABLE table=default.Table_2f9fb44984074b81baaac14bf2486c1c [id=2677b3a7e24a4bf8b897583cf2f94a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.141799 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72c9da889485fa46293a4651fe120 (DEFAULT_TABLE table=default.Table_f9c98f7833384e2cbb77a5a16f0bc77c [id=155cfe80917f40e78f5dc76687bf9387]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.142217 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72c9da889485fa46293a4651fe120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.145381 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1fc5c43794b0b8cf4d9650fcadef2 (DEFAULT_TABLE table=default.Table_2f9fb44984074b81baaac14bf2486c1c [id=2677b3a7e24a4bf8b897583cf2f94a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.145893 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1fc5c43794b0b8cf4d9650fcadef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.147836 6830 leader_election.cc:288] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.148531 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:46.148695 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:46.148902 4812 raft_consensus.cc:3037] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:46.149420 4608 leader_election.cc:341] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.151913 4812 raft_consensus.cc:2454] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:46.152863 4608 leader_election.cc:302] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.153820 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72c9da889485fa46293a4651fe120 (DEFAULT_TABLE table=default.Table_f9c98f7833384e2cbb77a5a16f0bc77c [id=155cfe80917f40e78f5dc76687bf9387]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.153928 6830 raft_consensus.cc:2781] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.154152 6830 raft_consensus.cc:686] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:46.154244 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72c9da889485fa46293a4651fe120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.154404 6830 consensus_queue.cc:237] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.121490 6841 raft_consensus.cc:504] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.157745 6841 leader_election.cc:288] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.158681 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0cf7a743e644d59e40694cc45cebaa (DEFAULT_TABLE table=default.Table_1ad83b282d514dc290b287c145589308 [id=7cfc78f272a04d8b9559b97fda0704e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.159179 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0cf7a743e644d59e40694cc45cebaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.160048 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 62a870f37ae1458e978307e0d3505711 (DEFAULT_TABLE table=default.Table_94b433051bd4452cbb4566bba7d627a3 [id=e0cef1fa41ed4f5392486882b9f13edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.160522 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a870f37ae1458e978307e0d3505711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.153834 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 919) took 3934 ms. Trace:
I20240503 02:12:46.162586 5159 rpcz_store.cc:276] 0503 02:12:42.219823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.219870 (+ 47us) service_pool.cc:225] Handling call
0503 02:12:46.070199 (+3850329us) ts_tablet_manager.cc:1120] Deleting tablet 810d37e454cb4633a099c546925eb48c
0503 02:12:46.090961 (+ 20762us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.153814 (+ 62853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3850179}
I20240503 02:12:46.163054 5159 tablet_replica.cc:330] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:46.163316 5159 raft_consensus.cc:2229] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.163578 5159 raft_consensus.cc:2258] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.164230 5159 ts_tablet_manager.cc:1875] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.165901 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 88b173d78ba74ede801b12cccbeef6b1 (table default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]) successfully deleted
I20240503 02:12:46.166620 6892 raft_consensus.cc:482] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.166772 6892 raft_consensus.cc:504] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.167004 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 42e80cc31f5c4908b82f2b030255f6af (table default.Table_be851867e140416c8921098eaa348201 [id=04d5d63e679c47ce9a72f1ffeeeaf506]) successfully deleted
I20240503 02:12:46.167455 6892 leader_election.cc:288] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.167682 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72c9da889485fa46293a4651fe120 (DEFAULT_TABLE table=default.Table_f9c98f7833384e2cbb77a5a16f0bc77c [id=155cfe80917f40e78f5dc76687bf9387]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.167908 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a870f37ae1458e978307e0d3505711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.168119 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72c9da889485fa46293a4651fe120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.171934 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a673151d0db649a0914141f7623d11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.173148 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3ca23bd0149d396950a340632898d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:46.176892 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b7d0640cea2a482e9974376ba479163b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d0640cea2a482e9974376ba479163b already in progress: creating tablet
I20240503 02:12:46.177135 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:46.177356 4462 leader_election.cc:341] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:46.177428 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1fc5c43794b0b8cf4d9650fcadef2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:46.178122 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 810d37e454cb4633a099c546925eb48c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810d37e454cb4633a099c546925eb48c already in progress: creating tablet
I20240503 02:12:46.179220 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 3db94fb9d53044d796e2973143d888fb (table default.Table_3e7dad4cf88048efa489e61c2595ad65 [id=5ca70228a70e4c02995e3c058a569e68]) successfully deleted
W20240503 02:12:46.180853 4608 leader_election.cc:341] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.181635 5114 ts_tablet_manager.cc:1888] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.181768 5114 log.cc:1196] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/76b01907833d44f6a000a4de3a2a6e96
I20240503 02:12:46.182190 5114 ts_tablet_manager.cc:1909] T 76b01907833d44f6a000a4de3a2a6e96 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.183117 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 907) took 2609 ms. Trace:
I20240503 02:12:46.183224 5114 rpcz_store.cc:276] 0503 02:12:43.573370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.573838 (+ 468us) service_pool.cc:225] Handling call
0503 02:12:46.066096 (+2492258us) ts_tablet_manager.cc:1120] Deleting tablet 76b01907833d44f6a000a4de3a2a6e96
0503 02:12:46.066112 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.130884 (+ 64772us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.181490 (+ 50606us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.183076 (+ 1586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2492093}
I20240503 02:12:46.184386 5103 tablet_bootstrap.cc:492] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.184545 5103 ts_tablet_manager.cc:1367] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.213s user 0.017s sys 0.072s
I20240503 02:12:46.185216 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 908) took 2611 ms. Trace:
I20240503 02:12:46.185321 5114 rpcz_store.cc:276] 0503 02:12:43.573467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.574220 (+ 753us) service_pool.cc:225] Handling call
0503 02:12:46.185054 (+2610834us) ts_tablet_manager.cc:1120] Deleting tablet ea7afb429d64497abcd9f1d0910651c3
0503 02:12:46.185069 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.185192 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2610697}
I20240503 02:12:46.185654 4266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_2bc261e8aa5a478786fbc5296356f0d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:46.186347 4266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bc261e8aa5a478786fbc5296356f0d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.187224 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b143c398ccbe42f09fa2e33f04f0d4d7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:46.187440 4812 raft_consensus.cc:3037] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.189752 5105 tablet_bootstrap.cc:492] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.189908 5105 ts_tablet_manager.cc:1367] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.005s
I20240503 02:12:46.191056 5103 raft_consensus.cc:348] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.191345 5103 raft_consensus.cc:374] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.191406 5103 raft_consensus.cc:729] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.191718 5103 consensus_queue.cc:260] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.192966 5103 ts_tablet_manager.cc:1398] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:46.192610 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 76b01907833d44f6a000a4de3a2a6e96 (table default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]) successfully deleted
W20240503 02:12:46.193691 4259 catalog_manager.cc:4797] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet ea7afb429d64497abcd9f1d0910651c3 because the tablet was not found. No further retry: Not found: Tablet not found: ea7afb429d64497abcd9f1d0910651c3
I20240503 02:12:46.194007 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d6d3ca80c54c629a5de82b06a7aac4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.194113 4274 master_service.cc:946] ReplaceTablet: received request to replace tablet d99a7e930a3947b4aae62424f11d7227 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:46.194236 4667 raft_consensus.cc:2454] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:46.194692 4462 leader_election.cc:302] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:46.194777 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_dbb9132d9f3441548f0e10b9818769f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:46.195436 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb9132d9f3441548f0e10b9818769f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.197691 6831 tablet.cc:1722] T e9f630bf6a5740758fa1f81477ec59d3 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:46.203629 6862 tablet.cc:1722] T e9f630bf6a5740758fa1f81477ec59d3 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:46.204842 4272 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 65ba631d85854b43bcb4bece9bfba7d3
I20240503 02:12:46.205770 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.205958 4812 raft_consensus.cc:2454] T b143c398ccbe42f09fa2e33f04f0d4d7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:46.206035 4269 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 620223977a5f48209f6cc12b53915508
I20240503 02:12:46.206418 6619 raft_consensus.cc:2781] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.206503 6619 raft_consensus.cc:482] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.206573 6619 raft_consensus.cc:3037] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.206851 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 620223977a5f48209f6cc12b53915508 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.206842 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_552f6a00a910447e9ee59068d1d3908f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:46.207438 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_552f6a00a910447e9ee59068d1d3908f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.207515 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 620223977a5f48209f6cc12b53915508 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.207782 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.208077 4608 leader_election.cc:302] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.208366 6841 raft_consensus.cc:2781] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.208461 6841 raft_consensus.cc:686] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:46.208743 4273 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 7a047360ed8843008c4ba633a65b2769
I20240503 02:12:46.208688 6841 consensus_queue.cc:237] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.209586 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_e8ca78c1991543479153a973a8346732" } new_table_name: "default.Table_ec81cfea38bf4719b2e30c82d4d93f50" modify_external_catalogs: true
I20240503 02:12:46.210301 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a047360ed8843008c4ba633a65b2769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.210956 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a047360ed8843008c4ba633a65b2769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.211004 4275 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 81213d7b9fc9429abd17f93b9f6982c1
I20240503 02:12:46.211706 4275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_6dd79385e5714ac1828bcf7463efbd5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:46.211890 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.212213 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0cf7a743e644d59e40694cc45cebaa (DEFAULT_TABLE table=default.Table_1ad83b282d514dc290b287c145589308 [id=7cfc78f272a04d8b9559b97fda0704e1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:46.212359 4275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dd79385e5714ac1828bcf7463efbd5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.212550 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.212656 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0cf7a743e644d59e40694cc45cebaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.212793 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 620223977a5f48209f6cc12b53915508 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.213207 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a047360ed8843008c4ba633a65b2769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.213953 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 81213d7b9fc9429abd17f93b9f6982c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:46.215289 4608 consensus_peers.cc:485] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76b01907833d44f6a000a4de3a2a6e96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:46.215682 4267 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:46.216080 4267 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:46.216562 4271 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet 557f1e3b3df64a8f899b08ba8dccc8f8
I20240503 02:12:46.217442 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_2393371f385c4869b2be72bd2dad35bf" } new_table_name: "default.Table_a24d81dbcfce407cad62bb25d8f6cad1" modify_external_catalogs: true
I20240503 02:12:46.217832 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.218163 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.223202 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ba631d85854b43bcb4bece9bfba7d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.223212 4272 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_62f0644508ee4e159ae0fd95cb3558c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:46.224009 4272 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62f0644508ee4e159ae0fd95cb3558c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.235016 5114 tablet_replica.cc:330] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.235251 5114 raft_consensus.cc:2229] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.235291 5103 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.235435 5114 raft_consensus.cc:2258] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.235965 5114 ts_tablet_manager.cc:1875] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.237476 4266 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_3b36c6850376429a8ac96f0a2595db44" } new_table_name: "default.Table_3072e66f7bb54c08999f93759967fa1a" modify_external_catalogs: true
I20240503 02:12:46.239607 4274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_e12eac6a8aaa48a9b27f9995e7e36b10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:46.240427 4274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e12eac6a8aaa48a9b27f9995e7e36b10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.241375 5115 ts_tablet_manager.cc:1888] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 02:12:46.247284 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5f8d331a718475a9f56dc7f3ec0e6b9 consensus state has no leader
W20240503 02:12:46.248327 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f4aad131b884c54b69b3d1d9420662c consensus state has no leader
I20240503 02:12:46.242046 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.248710 5115 log.cc:1196] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/d39f042afb954daa923502ef12377bd5
I20240503 02:12:46.249151 5115 ts_tablet_manager.cc:1909] T d39f042afb954daa923502ef12377bd5 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.249338 6857 tablet.cc:1722] T e9f630bf6a5740758fa1f81477ec59d3 P f4f9f079230a468f90e5e3a58f254b91: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:46.249496 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 815) took 5135 ms. Trace:
I20240503 02:12:46.249581 5115 rpcz_store.cc:276] 0503 02:12:41.113981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.114035 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:46.095087 (+4981052us) ts_tablet_manager.cc:1120] Deleting tablet d39f042afb954daa923502ef12377bd5
0503 02:12:46.095097 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.150405 (+ 55308us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.241239 (+ 90834us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.249476 (+ 8237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4980856}
I20240503 02:12:46.251315 6619 raft_consensus.cc:504] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.257088 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45703baa79ea4aaa8d9181bc25480b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.257930 6619 leader_election.cc:288] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:46.258209 4462 leader_election.cc:341] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:46.258401 5115 tablet_replica.cc:330] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.258617 5115 raft_consensus.cc:2229] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.258775 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0cf7a743e644d59e40694cc45cebaa (DEFAULT_TABLE table=default.Table_1ad83b282d514dc290b287c145589308 [id=7cfc78f272a04d8b9559b97fda0704e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.258999 5115 raft_consensus.cc:2258] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.259824 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet d39f042afb954daa923502ef12377bd5 (table default.Table_2c388fbe8fd148dc93f0d292381c0363 [id=8f46a18b9cdb40ac9d6c32ab3bf3424b]) successfully deleted
W20240503 02:12:46.261163 6896 log.cc:924] Time spent T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240503 02:12:46.265954 5103 tablet_bootstrap.cc:654] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.279731 5115 ts_tablet_manager.cc:1875] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.280280 5105 raft_consensus.cc:348] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.264889 6899 raft_consensus.cc:482] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.280501 5105 raft_consensus.cc:374] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.280561 5105 raft_consensus.cc:729] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.280800 5105 consensus_queue.cc:260] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.281440 5105 ts_tablet_manager.cc:1398] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20240503 02:12:46.281764 5105 tablet_bootstrap.cc:492] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.283613 5105 tablet_bootstrap.cc:654] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.284003 5159 ts_tablet_manager.cc:1888] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:46.284160 5159 log.cc:1196] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/64cdb861bc9f448698e243718136c303
I20240503 02:12:46.284631 5159 ts_tablet_manager.cc:1909] T 64cdb861bc9f448698e243718136c303 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:46.285212 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0cf7a743e644d59e40694cc45cebaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.285568 6899 raft_consensus.cc:504] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.288770 5114 ts_tablet_manager.cc:1888] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.288944 5114 log.cc:1196] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/cd66f88a24b74f2c9b2050af081d4601
I20240503 02:12:46.289319 5114 ts_tablet_manager.cc:1909] T cd66f88a24b74f2c9b2050af081d4601 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.289341 6899 leader_election.cc:288] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.290166 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 911) took 2672 ms. Trace:
I20240503 02:12:46.290305 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.290268 5114 rpcz_store.cc:276] 0503 02:12:43.618092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.618656 (+ 564us) service_pool.cc:225] Handling call
0503 02:12:46.189905 (+2571249us) ts_tablet_manager.cc:1120] Deleting tablet cd66f88a24b74f2c9b2050af081d4601
0503 02:12:46.234795 (+ 44890us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.244941 (+ 10146us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.288640 (+ 43699us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.290139 (+ 1499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2571102}
I20240503 02:12:46.291360 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet cd66f88a24b74f2c9b2050af081d4601 (table default.Table_3d65228f6c2b4694bb60f3118ad65af7 [id=2d5bfb965c844a199fa81636be8265a6]) successfully deleted
I20240503 02:12:46.292307 5106 tablet_bootstrap.cc:492] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.292534 5106 ts_tablet_manager.cc:1367] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.273s user 0.001s sys 0.005s
I20240503 02:12:46.296241 4271 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_15da6a828615427d9059c958249befbc" } new_table_name: "default.Table_54290c29045e4307949ba480b2eefeef" modify_external_catalogs: true
I20240503 02:12:46.297369 4269 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36560:
name: "default.Table_370e31b9745e49a69e323d187f951fc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:46.297775 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 912) took 2678 ms. Trace:
W20240503 02:12:46.297986 4269 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_370e31b9745e49a69e323d187f951fc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:46.297890 5114 rpcz_store.cc:276] 0503 02:12:43.619148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.619199 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:46.297536 (+2678337us) ts_tablet_manager.cc:1120] Deleting tablet 0794df2bfc5e45db95cf4c745892e695
0503 02:12:46.297554 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.297743 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678175}
I20240503 02:12:46.298631 5114 tablet_replica.cc:330] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.298885 5114 raft_consensus.cc:2229] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.299265 5114 raft_consensus.cc:2258] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:46.299507 4608 leader_election.cc:341] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.299798 4273 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36560:
table { table_name: "default.Table_23b17ecf55e548ff9ab71cd49b7d6fb1" } new_table_name: "default.Table_b93e282814624167a59c0ee3582f3b3d" modify_external_catalogs: true
W20240503 02:12:46.300011 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 0794df2bfc5e45db95cf4c745892e695 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0794df2bfc5e45db95cf4c745892e695 already in progress: creating tablet
I20240503 02:12:46.302911 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.304548 4812 raft_consensus.cc:2454] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.305593 4608 leader_election.cc:302] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.307754 5103 tablet_bootstrap.cc:492] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.308017 5103 ts_tablet_manager.cc:1367] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.000s
I20240503 02:12:46.310703 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d6d3ca80c54c629a5de82b06a7aac4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:46.310928 4667 raft_consensus.cc:3037] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.314800 5105 tablet_bootstrap.cc:492] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.315017 5105 ts_tablet_manager.cc:1367] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
W20240503 02:12:46.315658 4266 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09acc937f55944e5adea8b2ce1a46922 consensus state has no leader
I20240503 02:12:46.315733 5105 raft_consensus.cc:348] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.315929 5105 raft_consensus.cc:374] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.315976 5105 raft_consensus.cc:729] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.316493 4667 raft_consensus.cc:2454] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:46.322603 5105 consensus_queue.cc:260] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.323467 5103 raft_consensus.cc:348] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:46.330600 4273 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f3a9ee3274d488b9f80abc651bd3a34 has no consensus state
W20240503 02:12:46.331130 4271 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a8740e2e0d443d88a2cbd624341edea consensus state has no leader
I20240503 02:12:46.317621 4462 leader_election.cc:302] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:46.334823 5103 raft_consensus.cc:374] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.334946 5103 raft_consensus.cc:729] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.335215 6619 raft_consensus.cc:2781] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.335340 6619 raft_consensus.cc:686] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:46.335558 6619 consensus_queue.cc:237] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.335788 5115 ts_tablet_manager.cc:1888] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.335924 5115 log.cc:1196] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/87713e899d2148308316d807a6d76921
I20240503 02:12:46.336758 5115 ts_tablet_manager.cc:1909] T 87713e899d2148308316d807a6d76921 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.342667 5105 ts_tablet_manager.cc:1398] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:12:46.343129 5105 tablet_bootstrap.cc:492] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.335223 5103 consensus_queue.cc:260] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.344274 5103 ts_tablet_manager.cc:1398] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:12:46.317018 5114 ts_tablet_manager.cc:1875] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.345467 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 816) took 5231 ms. Trace:
I20240503 02:12:46.345608 5115 rpcz_store.cc:276] 0503 02:12:41.114485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.114530 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:46.258344 (+5143814us) ts_tablet_manager.cc:1120] Deleting tablet 87713e899d2148308316d807a6d76921
0503 02:12:46.258354 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.308449 (+ 50095us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.335667 (+ 27218us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.345435 (+ 9768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5143654}
I20240503 02:12:46.346040 5115 tablet_replica.cc:330] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.346210 5103 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.346302 5115 raft_consensus.cc:2229] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.346760 5115 raft_consensus.cc:2258] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.351032 5105 tablet_bootstrap.cc:654] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.357410 5103 tablet_bootstrap.cc:654] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.347528 5115 ts_tablet_manager.cc:1875] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.348101 5106 raft_consensus.cc:348] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.360615 5106 raft_consensus.cc:374] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.360754 5106 raft_consensus.cc:729] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.361119 5106 consensus_queue.cc:260] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.361948 5106 ts_tablet_manager.cc:1398] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20240503 02:12:46.362350 5106 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.364068 5106 tablet_bootstrap.cc:654] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.374874 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet e9f630bf6a5740758fa1f81477ec59d3 (table default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6]). Expected version 2 got 0
I20240503 02:12:46.375425 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_9091ef41bcba4103811266c2b3782904 [id=fde711a2c8e04377996a7cd87f629e68]). Expected version 2 got 0
I20240503 02:12:46.375617 4270 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:46.376248 5103 tablet_bootstrap.cc:492] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.376384 4267 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_b145550ee4dc4b6a8f00b076f2ac72a5 [id=2817ff73927a4593bddef5408db5cc49] alter complete (version 1)
I20240503 02:12:46.377111 5103 ts_tablet_manager.cc:1367] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.002s
I20240503 02:12:46.377967 5103 raft_consensus.cc:348] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.386638 5103 raft_consensus.cc:374] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.387087 5103 raft_consensus.cc:729] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.387579 5103 consensus_queue.cc:260] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.380474 5114 ts_tablet_manager.cc:1888] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.389885 5114 log.cc:1196] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:46.390543 5114 ts_tablet_manager.cc:1909] T b38d63d2cc8c4f54ba43cdf8cde1841f P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.391173 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d6d3ca80c54c629a5de82b06a7aac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:38 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.391351 4498 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c13f257eae44689af07cc31ea254c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.391999 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 910) took 2774 ms. Trace:
I20240503 02:12:46.392076 5114 rpcz_store.cc:276] 0503 02:12:43.617960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.618154 (+ 194us) service_pool.cc:225] Handling call
0503 02:12:46.298552 (+2680398us) ts_tablet_manager.cc:1120] Deleting tablet b38d63d2cc8c4f54ba43cdf8cde1841f
0503 02:12:46.298565 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.366826 (+ 68261us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.380339 (+ 13513us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.391979 (+ 11640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676402}
I20240503 02:12:46.389578 5103 ts_tablet_manager.cc:1398] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:12:46.396099 5115 ts_tablet_manager.cc:1888] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.390990 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce11c70a924b8d92d8f36fc7401dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:22 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.399586 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet b38d63d2cc8c4f54ba43cdf8cde1841f (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]) successfully deleted
I20240503 02:12:46.400169 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 5984a3757146438d8b06a65f74cd3d6d (DEFAULT_TABLE table=default.Table_d6cf353d37364d2098437daadf566b51 [id=2a57ed99efe34fcb88f092d10589ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.400652 4495 tablet_service.cc:1467] Processing CreateTablet for tablet f087a02dd90b4f48833b3e98204cd9d6 (DEFAULT_TABLE table=default.Table_997cf815690744c3a6c9732ff0802a5e [id=8c7c910682db42d1bf8fbb204ec6ca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.401074 4503 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a9ee3274d488b9f80abc651bd3a34 (DEFAULT_TABLE table=default.Table_b93e282814624167a59c0ee3582f3b3d [id=aa2c745e913549bcb711565f23daa29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.401466 4498 tablet_service.cc:1467] Processing CreateTablet for tablet 8688fd5cd2bb495eb2558cd247db931d (DEFAULT_TABLE table=default.Table_f028f0dfeea1433ca95764504d64c85e [id=16e41e7059554b6089ce860181ff6ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.401479 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 920) took 4103 ms. Trace:
I20240503 02:12:46.401587 5159 rpcz_store.cc:276] 0503 02:12:42.298106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.298163 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:46.163000 (+3864837us) ts_tablet_manager.cc:1120] Deleting tablet 64cdb861bc9f448698e243718136c303
0503 02:12:46.163014 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.268545 (+105531us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.283826 (+ 15281us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.401437 (+117611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3864661}
I20240503 02:12:46.401988 5159 tablet_replica.cc:330] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:46.402096 4496 tablet_service.cc:1467] Processing CreateTablet for tablet f00221dacba3413a9bfc9134b8b902f7 (DEFAULT_TABLE table=default.Table_c9320e1ce9bc491faf424b7db2850839 [id=b57c1c16607f4f208fb915c06079f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.402204 5159 raft_consensus.cc:2229] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.402356 5159 raft_consensus.cc:2258] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.402937 5159 ts_tablet_manager.cc:1875] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.402909 4507 tablet_service.cc:1467] Processing CreateTablet for tablet f951c8a82e9545b09779726e96f2d2ec (DEFAULT_TABLE table=default.Table_5bce1cbd35204706ab1c70da3a37d8dd [id=cc8ee641bf614ce1a3ec28f3bd5ec424]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.402980 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 930) took 2377 ms. Trace:
I20240503 02:12:46.403082 5114 rpcz_store.cc:276] 0503 02:12:44.025949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.026058 (+ 109us) service_pool.cc:225] Handling call
0503 02:12:46.392362 (+2366304us) ts_tablet_manager.cc:1120] Deleting tablet 151f925dcb304594b60c85d850b83533
0503 02:12:46.392371 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.402965 (+ 10594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2365899}
I20240503 02:12:46.406039 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]) successfully deleted
I20240503 02:12:46.407052 5103 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.408875 5103 tablet_bootstrap.cc:654] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.409209 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5984a3757146438d8b06a65f74cd3d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.410526 4268 master_service.cc:946] ReplaceTablet: received request to replace tablet e4cb6432296c4a45850a96c3651ab7bd from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:46.410705 4165 master-stress-test.cc:578] Killing test thread 1/25
I20240503 02:12:46.413450 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f087a02dd90b4f48833b3e98204cd9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.417093 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a9ee3274d488b9f80abc651bd3a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.418722 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688fd5cd2bb495eb2558cd247db931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.419768 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00221dacba3413a9bfc9134b8b902f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.421802 4165 master-stress-test.cc:580] Killed test thread 1/25
I20240503 02:12:46.421911 4165 master-stress-test.cc:578] Killing test thread 2/25
I20240503 02:12:46.421990 4165 master-stress-test.cc:580] Killed test thread 2/25
I20240503 02:12:46.422035 4165 master-stress-test.cc:578] Killing test thread 3/25
I20240503 02:12:46.422111 4165 master-stress-test.cc:580] Killed test thread 3/25
I20240503 02:12:46.422155 4165 master-stress-test.cc:578] Killing test thread 4/25
I20240503 02:12:46.422236 4165 master-stress-test.cc:580] Killed test thread 4/25
I20240503 02:12:46.422281 4165 master-stress-test.cc:578] Killing test thread 5/25
I20240503 02:12:46.422355 4165 master-stress-test.cc:580] Killed test thread 5/25
I20240503 02:12:46.422400 4165 master-stress-test.cc:578] Killing test thread 6/25
I20240503 02:12:46.422473 4165 master-stress-test.cc:580] Killed test thread 6/25
I20240503 02:12:46.422520 4165 master-stress-test.cc:578] Killing test thread 7/25
I20240503 02:12:46.422595 4165 master-stress-test.cc:580] Killed test thread 7/25
I20240503 02:12:46.422642 4165 master-stress-test.cc:578] Killing test thread 8/25
I20240503 02:12:46.422719 4165 master-stress-test.cc:580] Killed test thread 8/25
I20240503 02:12:46.422768 4165 master-stress-test.cc:578] Killing test thread 9/25
I20240503 02:12:46.422848 4165 master-stress-test.cc:580] Killed test thread 9/25
I20240503 02:12:46.422894 4165 master-stress-test.cc:578] Killing test thread 10/25
I20240503 02:12:46.423015 5103 tablet_bootstrap.cc:492] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.423146 5103 ts_tablet_manager.cc:1367] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240503 02:12:46.423219 4165 master-stress-test.cc:580] Killed test thread 10/25
I20240503 02:12:46.423288 4165 master-stress-test.cc:578] Killing test thread 11/25
I20240503 02:12:46.423369 4165 master-stress-test.cc:580] Killed test thread 11/25
I20240503 02:12:46.423420 4165 master-stress-test.cc:578] Killing test thread 12/25
I20240503 02:12:46.423493 4165 master-stress-test.cc:580] Killed test thread 12/25
I20240503 02:12:46.423542 4165 master-stress-test.cc:578] Killing test thread 13/25
I20240503 02:12:46.423616 4165 master-stress-test.cc:580] Killed test thread 13/25
I20240503 02:12:46.423666 4165 master-stress-test.cc:578] Killing test thread 14/25
I20240503 02:12:46.423739 4165 master-stress-test.cc:580] Killed test thread 14/25
I20240503 02:12:46.423789 4165 master-stress-test.cc:578] Killing test thread 15/25
I20240503 02:12:46.423862 4165 master-stress-test.cc:580] Killed test thread 15/25
I20240503 02:12:46.423913 4165 master-stress-test.cc:578] Killing test thread 16/25
I20240503 02:12:46.423846 5103 raft_consensus.cc:348] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.423985 4165 master-stress-test.cc:580] Killed test thread 16/25
I20240503 02:12:46.424032 4165 master-stress-test.cc:578] Killing test thread 17/25
I20240503 02:12:46.424026 5103 raft_consensus.cc:374] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.424072 5103 raft_consensus.cc:729] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.424108 4165 master-stress-test.cc:580] Killed test thread 17/25
I20240503 02:12:46.424158 4165 master-stress-test.cc:578] Killing test thread 18/25
I20240503 02:12:46.424245 4165 master-stress-test.cc:580] Killed test thread 18/25
I20240503 02:12:46.424296 4165 master-stress-test.cc:578] Killing test thread 19/25
I20240503 02:12:46.424382 4165 master-stress-test.cc:580] Killed test thread 19/25
I20240503 02:12:46.424257 5103 consensus_queue.cc:260] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.424479 4165 master-stress-test.cc:578] Killing test thread 20/25
I20240503 02:12:46.424589 4165 master-stress-test.cc:580] Killed test thread 20/25
I20240503 02:12:46.424667 4165 master-stress-test.cc:578] Killing test thread 21/25
I20240503 02:12:46.424765 5103 ts_tablet_manager.cc:1398] T b143c398ccbe42f09fa2e33f04f0d4d7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:46.425077 5103 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.426972 5103 tablet_bootstrap.cc:654] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.427682 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 931) took 2395 ms. Trace:
I20240503 02:12:46.427814 5114 rpcz_store.cc:276] 0503 02:12:44.031798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.031905 (+ 107us) service_pool.cc:225] Handling call
0503 02:12:46.403436 (+2371531us) ts_tablet_manager.cc:1120] Deleting tablet e82190dfb1b1418b9f20bf4d0c2c2409
0503 02:12:46.427515 (+ 24079us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.427659 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2370855}
I20240503 02:12:46.428225 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 932) took 2390 ms. Trace:
I20240503 02:12:46.428299 5114 rpcz_store.cc:276] 0503 02:12:44.037949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.038007 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:46.428079 (+2390072us) ts_tablet_manager.cc:1120] Deleting tablet 3ee32345fb9349f19af09a3447c4caad
0503 02:12:46.428089 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.428208 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389794}
W20240503 02:12:46.429055 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 151f925dcb304594b60c85d850b83533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 151f925dcb304594b60c85d850b83533 already in progress: creating tablet
I20240503 02:12:46.401814 5106 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
W20240503 02:12:46.429431 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet e82190dfb1b1418b9f20bf4d0c2c2409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82190dfb1b1418b9f20bf4d0c2c2409 already in progress: creating tablet
W20240503 02:12:46.429680 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 3ee32345fb9349f19af09a3447c4caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee32345fb9349f19af09a3447c4caad already in progress: creating tablet
I20240503 02:12:46.429703 5106 ts_tablet_manager.cc:1367] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.001s
I20240503 02:12:46.430513 5106 raft_consensus.cc:348] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.430814 5106 raft_consensus.cc:374] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.430928 5106 raft_consensus.cc:729] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.431205 5106 consensus_queue.cc:260] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.431969 5106 ts_tablet_manager.cc:1398] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:46.432149 5115 log.cc:1196] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/ea5e4db5204e4acf8719100d02d533fb
I20240503 02:12:46.428609 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 933) took 2384 ms. Trace:
I20240503 02:12:46.429184 4494 tablet_service.cc:1467] Processing CreateTablet for tablet b9fd946ea2cb481db7506bdfc834dd4a (DEFAULT_TABLE table=default.Table_3f411ccab7cf48818684560067793bfa [id=89eb739dc9b0477eb4d9c38b876a03b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.435369 5114 rpcz_store.cc:276] 0503 02:12:44.044084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.044141 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:46.428508 (+2384367us) ts_tablet_manager.cc:1120] Deleting tablet 2848c837598c49628c11374d3c875b3e
0503 02:12:46.428517 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.428595 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2384161}
I20240503 02:12:46.435696 5114 tablet_replica.cc:330] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.429216 4493 tablet_service.cc:1467] Processing CreateTablet for tablet d757bcb48d1042a78b439a31e5b5c86b (DEFAULT_TABLE table=default.Table_b7257d81b4094fc495ca99d263034757 [id=5d8525a7bebb4ef083530aedae96ad64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.435874 5114 raft_consensus.cc:2229] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.436056 5114 raft_consensus.cc:2258] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.428594 6769 raft_consensus.cc:504] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.429510 4491 tablet_service.cc:1467] Processing CreateTablet for tablet f23c97f74e2a4981add1f663b04f40c5 (DEFAULT_TABLE table=default.Table_ca7d5d73ff93496b90bdb5aebca4cde5 [id=3b8ca214ab0a49d4afe2a8443214904b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.429656 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 379b80337ec54489b0d7dda1df420c7f (DEFAULT_TABLE table=default.Table_2c2f293e071545c7ae06e32986fbf729 [id=649a464d2c964714a72528b978f0f9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.437579 5114 ts_tablet_manager.cc:1875] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.427682 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951c8a82e9545b09779726e96f2d2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.438041 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d757bcb48d1042a78b439a31e5b5c86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.433774 4500 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1649e2c9a64f81a58ecb4df695dc8a (DEFAULT_TABLE table=default.Table_29b8558332544b49aefeb049f9452eb8 [id=ecba57e671f549eda5f0894494f3d04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.441183 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1649e2c9a64f81a58ecb4df695dc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.442364 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fd946ea2cb481db7506bdfc834dd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.434464 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 3570d8b95daa4a18b2b0a69af627e7a4 (DEFAULT_TABLE table=default.Table_fbe14bb502bb465a95363bfcbbded773 [id=7fba00e5678944789de68d6dac6cbdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.443421 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570d8b95daa4a18b2b0a69af627e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.435029 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 30b003a2fc5c440aafa64242a3fa1adc (DEFAULT_TABLE table=default.Table_2b1d96d5b6c942f0914e000795c0386f [id=a088ed5d80b845ebbbee25e5a31acbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.428969 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 07cca97600374070b117e3ef3869042d (DEFAULT_TABLE table=default.Table_2acba999c8d0432caf7f1cc0f404b8d0 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.445096 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b003a2fc5c440aafa64242a3fa1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.445243 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07cca97600374070b117e3ef3869042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.445852 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379b80337ec54489b0d7dda1df420c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.429342 4509 tablet_service.cc:1467] Processing CreateTablet for tablet 36c2c3983b0246a38be6d5be78aa3e5f (DEFAULT_TABLE table=default.Table_d2eeb6ba099049a1845ced64b6119f4d [id=0de9ca3926fc478f8f86ca0bd5032d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.446091 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c2c3983b0246a38be6d5be78aa3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.437022 6769 leader_election.cc:288] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.437813 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c13f257eae44689af07cc31ea254c4" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:46.451741 4667 raft_consensus.cc:3037] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.452984 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23c97f74e2a4981add1f663b04f40c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.454471 4667 raft_consensus.cc:2454] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:46.454937 4462 leader_election.cc:302] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
W20240503 02:12:46.457397 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 2848c837598c49628c11374d3c875b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2848c837598c49628c11374d3c875b3e already in progress: creating tablet
W20240503 02:12:46.461599 4462 consensus_peers.cc:485] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53b149e66a0446eeade190dd7f5d2bec. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:46.461835 4462 leader_election.cc:341] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 27c13f257eae44689af07cc31ea254c4
I20240503 02:12:46.462551 6892 raft_consensus.cc:2781] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.462674 6892 raft_consensus.cc:686] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:46.463017 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 8688fd5cd2bb495eb2558cd247db931d (DEFAULT_TABLE table=default.Table_f028f0dfeea1433ca95764504d64c85e [id=16e41e7059554b6089ce860181ff6ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.463451 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688fd5cd2bb495eb2558cd247db931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.464629 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 5984a3757146438d8b06a65f74cd3d6d (DEFAULT_TABLE table=default.Table_d6cf353d37364d2098437daadf566b51 [id=2a57ed99efe34fcb88f092d10589ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.465025 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5984a3757146438d8b06a65f74cd3d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.465176 5114 ts_tablet_manager.cc:1888] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.465325 5114 log.cc:1196] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/79b8db679ba24ac6a93be591c0513ab0
I20240503 02:12:46.465745 5114 ts_tablet_manager.cc:1909] T 79b8db679ba24ac6a93be591c0513ab0 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.466084 4789 tablet_service.cc:1467] Processing CreateTablet for tablet f00221dacba3413a9bfc9134b8b902f7 (DEFAULT_TABLE table=default.Table_c9320e1ce9bc491faf424b7db2850839 [id=b57c1c16607f4f208fb915c06079f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.466652 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 935) took 2404 ms. Trace:
I20240503 02:12:46.466773 4776 tablet_service.cc:1467] Processing CreateTablet for tablet f087a02dd90b4f48833b3e98204cd9d6 (DEFAULT_TABLE table=default.Table_997cf815690744c3a6c9732ff0802a5e [id=8c7c910682db42d1bf8fbb204ec6ca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.466745 5114 rpcz_store.cc:276] 0503 02:12:44.061768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.062645 (+ 877us) service_pool.cc:225] Handling call
0503 02:12:46.435653 (+2373008us) ts_tablet_manager.cc:1120] Deleting tablet 79b8db679ba24ac6a93be591c0513ab0
0503 02:12:46.435663 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.458975 (+ 23312us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.465057 (+ 6082us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.466628 (+ 1571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372807}
I20240503 02:12:46.467074 5114 tablet_replica.cc:330] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.466708 4267 catalog_manager.cc:5484] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.467269 5114 raft_consensus.cc:2229] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.467303 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet efa7c0c5fc614c16985008b8fba54bab (table default.Table_637aa20655eb485f921eeebcadeee3eb [id=475bf67c3074494eb43e612a6897f57b]). Expected version 2 got 0
I20240503 02:12:46.467587 5114 raft_consensus.cc:2258] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.467738 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00221dacba3413a9bfc9134b8b902f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.451354 5106 tablet_bootstrap.cc:492] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.469322 5106 tablet_bootstrap.cc:654] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.469411 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a9ee3274d488b9f80abc651bd3a34 (DEFAULT_TABLE table=default.Table_b93e282814624167a59c0ee3582f3b3d [id=aa2c745e913549bcb711565f23daa29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.469084 4267 catalog_manager.cc:5484] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.469884 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f087a02dd90b4f48833b3e98204cd9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.469870 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:46.470551 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a9ee3274d488b9f80abc651bd3a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.435267 5115 ts_tablet_manager.cc:1909] T ea5e4db5204e4acf8719100d02d533fb P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.470942 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 817) took 5349 ms. Trace:
I20240503 02:12:46.471081 5105 tablet_bootstrap.cc:492] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.471213 5105 ts_tablet_manager.cc:1367] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.010s
I20240503 02:12:46.471060 5115 rpcz_store.cc:276] 0503 02:12:41.121297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.121368 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:46.345980 (+5224612us) ts_tablet_manager.cc:1120] Deleting tablet ea5e4db5204e4acf8719100d02d533fb
0503 02:12:46.345991 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.378865 (+ 32874us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.395973 (+ 17108us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.470919 (+ 74946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5224396}
I20240503 02:12:46.471513 5115 tablet_replica.cc:330] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.471071 4267 catalog_manager.cc:5484] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.471643 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 07cca97600374070b117e3ef3869042d (DEFAULT_TABLE table=default.Table_2acba999c8d0432caf7f1cc0f404b8d0 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.471745 5115 raft_consensus.cc:2229] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.471908 5115 raft_consensus.cc:2258] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.471956 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:46.472146 5105 raft_consensus.cc:348] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.472350 5105 raft_consensus.cc:374] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.472419 5105 raft_consensus.cc:729] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.471644 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 79b8db679ba24ac6a93be591c0513ab0 (table default.Table_95c5dbfc7e2542dd93ea5839f66200e1 [id=ffbb2c8938d54556bf18d766e0c901cc]) successfully deleted
I20240503 02:12:46.472515 5115 ts_tablet_manager.cc:1875] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.472982 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1975c790bfa4449baa6d7b81509c5003 (table default.Table_1a50f808193c474c97646ae5b9ab2a0c [id=11645209f14b4d8880c3ca53263d4ec5]). Expected version 1 got 0
I20240503 02:12:46.473126 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07cca97600374070b117e3ef3869042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.473512 5105 consensus_queue.cc:260] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.474050 4780 tablet_service.cc:1467] Processing CreateTablet for tablet d757bcb48d1042a78b439a31e5b5c86b (DEFAULT_TABLE table=default.Table_b7257d81b4094fc495ca99d263034757 [id=5d8525a7bebb4ef083530aedae96ad64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.474406 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d757bcb48d1042a78b439a31e5b5c86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.473682 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc]). Expected version 2 got 0
I20240503 02:12:46.474982 4781 tablet_service.cc:1467] Processing CreateTablet for tablet f951c8a82e9545b09779726e96f2d2ec (DEFAULT_TABLE table=default.Table_5bce1cbd35204706ab1c70da3a37d8dd [id=cc8ee641bf614ce1a3ec28f3bd5ec424]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.475163 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:46.475420 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951c8a82e9545b09779726e96f2d2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.477145 5105 ts_tablet_manager.cc:1398] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:46.478080 5105 tablet_bootstrap.cc:492] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.486652 6892 consensus_queue.cc:237] T 27c13f257eae44689af07cc31ea254c4 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.474771 4790 tablet_service.cc:1467] Processing CreateTablet for tablet b9fd946ea2cb481db7506bdfc834dd4a (DEFAULT_TABLE table=default.Table_3f411ccab7cf48818684560067793bfa [id=89eb739dc9b0477eb4d9c38b876a03b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.498476 5114 ts_tablet_manager.cc:1875] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.499289 4778 tablet_service.cc:1467] Processing CreateTablet for tablet 379b80337ec54489b0d7dda1df420c7f (DEFAULT_TABLE table=default.Table_2c2f293e071545c7ae06e32986fbf729 [id=649a464d2c964714a72528b978f0f9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.499496 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 3570d8b95daa4a18b2b0a69af627e7a4 (DEFAULT_TABLE table=default.Table_fbe14bb502bb465a95363bfcbbded773 [id=7fba00e5678944789de68d6dac6cbdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.499745 4788 tablet_service.cc:1467] Processing CreateTablet for tablet f23c97f74e2a4981add1f663b04f40c5 (DEFAULT_TABLE table=default.Table_ca7d5d73ff93496b90bdb5aebca4cde5 [id=3b8ca214ab0a49d4afe2a8443214904b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.500142 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1649e2c9a64f81a58ecb4df695dc8a (DEFAULT_TABLE table=default.Table_29b8558332544b49aefeb049f9452eb8 [id=ecba57e671f549eda5f0894494f3d04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.498178 5103 tablet_bootstrap.cc:492] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.500386 5103 ts_tablet_manager.cc:1367] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.015s
I20240503 02:12:46.500492 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 30b003a2fc5c440aafa64242a3fa1adc (DEFAULT_TABLE table=default.Table_2b1d96d5b6c942f0914e000795c0386f [id=a088ed5d80b845ebbbee25e5a31acbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.500734 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 36c2c3983b0246a38be6d5be78aa3e5f (DEFAULT_TABLE table=default.Table_d2eeb6ba099049a1845ced64b6119f4d [id=0de9ca3926fc478f8f86ca0bd5032d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.501411 5103 raft_consensus.cc:348] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.501705 5103 raft_consensus.cc:374] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.501802 5103 raft_consensus.cc:729] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.502048 5103 consensus_queue.cc:260] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.502475 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fd946ea2cb481db7506bdfc834dd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.502652 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570d8b95daa4a18b2b0a69af627e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.502781 5103 ts_tablet_manager.cc:1398] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:46.502848 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379b80337ec54489b0d7dda1df420c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.503157 5103 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.503341 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23c97f74e2a4981add1f663b04f40c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.503805 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1649e2c9a64f81a58ecb4df695dc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.504201 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c2c3983b0246a38be6d5be78aa3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.509131 6897 raft_consensus.cc:482] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.509387 6897 raft_consensus.cc:504] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.510138 6897 leader_election.cc:288] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.511428 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b003a2fc5c440aafa64242a3fa1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.511816 5106 tablet_bootstrap.cc:492] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.512048 5106 ts_tablet_manager.cc:1367] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240503 02:12:46.512238 5103 tablet_bootstrap.cc:654] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.515733 5105 tablet_bootstrap.cc:654] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.516954 5106 raft_consensus.cc:348] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.517602 5106 raft_consensus.cc:374] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.517719 5106 raft_consensus.cc:729] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.518010 5106 consensus_queue.cc:260] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.523305 4270 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_76d4f904da56406595b0e3da677e7cc8 [id=4d9a163c0c564955a47a65f799a0018c] alter complete (version 6)
I20240503 02:12:46.526198 5106 ts_tablet_manager.cc:1398] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:46.526569 5106 tablet_bootstrap.cc:492] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.535926 4268 catalog_manager.cc:5771] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: Sending DeleteTablet for 3 replicas of tablet e4cb6432296c4a45850a96c3651ab7bd
I20240503 02:12:46.536602 4268 catalog_manager.cc:6554] ReplaceTablet: tablet e4cb6432296c4a45850a96c3651ab7bd deleted and replaced by tablet c70d86c941f2445fbd6ff8b0be7e4043
I20240503 02:12:46.538214 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cb6432296c4a45850a96c3651ab7bd with delete_type TABLET_DATA_DELETED (replaced by tablet c70d86c941f2445fbd6ff8b0be7e4043) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.540211 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_0187a6fc086f4da1a49978fae202c36c [id=48dba2f59b8d4023849a605c2346dfb6] alter complete (version 2)
I20240503 02:12:46.545814 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.546001 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.546281 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.546434 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:46.546710 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 04d555c8d8844ee49c960a5279485480 from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:46.528092 5106 tablet_bootstrap.cc:654] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.547766 6774 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: acquiring CA information for follower catalog manager: success
I20240503 02:12:46.548887 5103 tablet_bootstrap.cc:492] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.549150 5103 ts_tablet_manager.cc:1367] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240503 02:12:46.550335 5106 tablet_bootstrap.cc:492] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.551286 6892 consensus_queue.cc:1035] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.552029 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67857b858c4719b498291c52b7aceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.552600 5106 ts_tablet_manager.cc:1367] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:12:46.553508 5106 raft_consensus.cc:348] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.554242 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:46.554531 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:46.561456 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet dee34be89bda450296b916163bb10c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.562927 5103 raft_consensus.cc:348] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.563283 5103 raft_consensus.cc:374] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.563380 5103 raft_consensus.cc:729] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.563633 5106 raft_consensus.cc:374] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.563735 5106 raft_consensus.cc:729] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.563716 5103 consensus_queue.cc:260] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.563969 5106 consensus_queue.cc:260] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.564651 5103 ts_tablet_manager.cc:1398] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:46.563035 5115 ts_tablet_manager.cc:1888] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.572181 5115 log.cc:1196] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/95681e67db1e4485876d530133e8a514
I20240503 02:12:46.572819 5115 ts_tablet_manager.cc:1909] T 95681e67db1e4485876d530133e8a514 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.573907 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 818) took 5452 ms. Trace:
I20240503 02:12:46.574095 5115 rpcz_store.cc:276] 0503 02:12:41.121913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.121963 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:46.471468 (+5349505us) ts_tablet_manager.cc:1120] Deleting tablet 95681e67db1e4485876d530133e8a514
0503 02:12:46.471478 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.534779 (+ 63301us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.562908 (+ 28129us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.573888 (+ 10980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5349338}
I20240503 02:12:46.574560 5115 tablet_replica.cc:330] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.565196 5103 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.591249 5103 tablet_bootstrap.cc:654] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.591696 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cb6432296c4a45850a96c3651ab7bd with delete_type TABLET_DATA_DELETED (replaced by tablet c70d86c941f2445fbd6ff8b0be7e4043) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.593653 5114 ts_tablet_manager.cc:1888] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.594097 5115 raft_consensus.cc:2229] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.594174 5106 ts_tablet_manager.cc:1398] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 02:12:46.594343 5115 raft_consensus.cc:2258] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.594990 5115 ts_tablet_manager.cc:1875] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.600652 5106 tablet_bootstrap.cc:492] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.600999 6897 raft_consensus.cc:2781] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.601086 6897 raft_consensus.cc:482] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.601150 6897 raft_consensus.cc:3037] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.603561 5115 ts_tablet_manager.cc:1888] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.603792 5115 log.cc:1196] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/48e713ee37d042309166b0f118e14986
I20240503 02:12:46.603667 6897 raft_consensus.cc:504] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:46.604140 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.67:35993, user_credentials={real_user=slave}} blocked reactor thread for 298560us
I20240503 02:12:46.604419 6897 leader_election.cc:288] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.604450 5105 tablet_bootstrap.cc:492] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.605384 5115 ts_tablet_manager.cc:1909] T 48e713ee37d042309166b0f118e14986 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.607685 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 819) took 5485 ms. Trace:
I20240503 02:12:46.609035 5105 ts_tablet_manager.cc:1367] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.006s
I20240503 02:12:46.602481 6892 raft_consensus.cc:482] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.609529 5115 rpcz_store.cc:276] 0503 02:12:41.122435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.122481 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:46.574503 (+5452022us) ts_tablet_manager.cc:1120] Deleting tablet 48e713ee37d042309166b0f118e14986
0503 02:12:46.574516 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.599395 (+ 24879us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.603443 (+ 4048us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.607659 (+ 4216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19230,"tablet-delete.queue_time_us":5451868}
I20240503 02:12:46.609591 6892 raft_consensus.cc:504] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.609992 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 820) took 5398 ms. Trace:
I20240503 02:12:46.610064 5115 rpcz_store.cc:276] 0503 02:12:41.211263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.211317 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:46.609851 (+5398534us) ts_tablet_manager.cc:1120] Deleting tablet 0c3b9a888b894897b7cc3cf08402febd
0503 02:12:46.609861 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.609974 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5398372}
I20240503 02:12:46.610391 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 822) took 5393 ms. Trace:
I20240503 02:12:46.610458 5115 rpcz_store.cc:276] 0503 02:12:41.216577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.217366 (+ 789us) service_pool.cc:225] Handling call
0503 02:12:46.610287 (+5392921us) ts_tablet_manager.cc:1120] Deleting tablet efd8e0353b5f4ce484ac81d646b5323d
0503 02:12:46.610297 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.610377 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5392744}
I20240503 02:12:46.610715 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 821) took 5394 ms. Trace:
I20240503 02:12:46.611465 6892 leader_election.cc:288] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.611631 5115 rpcz_store.cc:276] 0503 02:12:41.216468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.217574 (+ 1106us) service_pool.cc:225] Handling call
0503 02:12:46.610636 (+5393062us) ts_tablet_manager.cc:1120] Deleting tablet 28c1a81589df496b92b980353f00e594
0503 02:12:46.610643 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.610703 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5392942}
I20240503 02:12:46.612035 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 823) took 5288 ms. Trace:
I20240503 02:12:46.612187 5115 rpcz_store.cc:276] 0503 02:12:41.323810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.323856 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:46.611935 (+5288079us) ts_tablet_manager.cc:1120] Deleting tablet 39c48b14bc624c28a871d45e9429ee30
0503 02:12:46.611944 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.612020 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5287911}
I20240503 02:12:46.612007 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.612653 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 824) took 5288 ms. Trace:
I20240503 02:12:46.613801 5114 log.cc:1196] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:46.614004 5115 rpcz_store.cc:276] 0503 02:12:41.324212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.324256 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:46.612549 (+5288293us) ts_tablet_manager.cc:1120] Deleting tablet 5f8656a886564f06931039b516162b51
0503 02:12:46.612559 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.612636 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5288081}
I20240503 02:12:46.614260 5114 ts_tablet_manager.cc:1909] T 035216c9475d4b59aa1ac81b285b4d8a P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.614607 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 826) took 5265 ms. Trace:
I20240503 02:12:46.614759 5115 rpcz_store.cc:276] 0503 02:12:41.349176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.349238 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:46.614463 (+5265225us) ts_tablet_manager.cc:1120] Deleting tablet 3c421694ce1e424da7953d061a6bca3e
0503 02:12:46.614477 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.614586 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5265018}
I20240503 02:12:46.615144 5115 tablet_replica.cc:330] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.615581 5115 raft_consensus.cc:2229] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.616779 5115 raft_consensus.cc:2258] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.617719 5115 ts_tablet_manager.cc:1875] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.618163 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 87713e899d2148308316d807a6d76921 (table default.Table_4702ebe6617c4aebb1264298076c6a4d [id=31669972ad634736b38958103d7b6c16]) successfully deleted
I20240503 02:12:46.626379 6769 raft_consensus.cc:482] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.626678 6769 raft_consensus.cc:504] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.627512 6769 leader_election.cc:288] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.627755 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 937) took 2468 ms. Trace:
I20240503 02:12:46.629331 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce11c70a924b8d92d8f36fc7401dfd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.618999 5106 tablet_bootstrap.cc:654] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.629583 4672 raft_consensus.cc:2454] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:46.630101 4462 leader_election.cc:302] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:46.634354 6769 raft_consensus.cc:2781] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.634471 6769 raft_consensus.cc:482] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.634539 6769 raft_consensus.cc:3037] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.634505 5114 rpcz_store.cc:276] 0503 02:12:44.159179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.159239 (+ 60us) service_pool.cc:225] Handling call
0503 02:12:46.467029 (+2307790us) ts_tablet_manager.cc:1120] Deleting tablet 035216c9475d4b59aa1ac81b285b4d8a
0503 02:12:46.467038 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.544039 (+ 77001us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.593527 (+ 49488us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.627737 (+ 34210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307335}
I20240503 02:12:46.634990 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 035216c9475d4b59aa1ac81b285b4d8a (table default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]) successfully deleted
I20240503 02:12:46.636504 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 936) took 2477 ms. Trace:
I20240503 02:12:46.636601 5114 rpcz_store.cc:276] 0503 02:12:44.159043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.159813 (+ 770us) service_pool.cc:225] Handling call
0503 02:12:46.635813 (+2476000us) ts_tablet_manager.cc:1120] Deleting tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2
0503 02:12:46.635827 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.636476 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2475813}
W20240503 02:12:46.637089 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 already in progress: creating tablet
I20240503 02:12:46.637425 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3dbcbc572f4215b7f59f3750a4b9fa" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.637588 5105 raft_consensus.cc:348] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.637743 4813 raft_consensus.cc:2379] T 1f3dbcbc572f4215b7f59f3750a4b9fa P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:46.637771 5105 raft_consensus.cc:374] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.637841 5105 raft_consensus.cc:729] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.638044 5114 tablet_replica.cc:330] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.638096 5105 consensus_queue.cc:260] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.638245 5114 raft_consensus.cc:2229] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.638419 5114 raft_consensus.cc:2258] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.638628 5105 ts_tablet_manager.cc:1398] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
I20240503 02:12:46.638906 5105 tablet_bootstrap.cc:492] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.638823 6769 raft_consensus.cc:504] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.639639 6769 leader_election.cc:288] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.640084 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce11c70a924b8d92d8f36fc7401dfd" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:46.640304 4672 raft_consensus.cc:3037] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.640962 5103 tablet_bootstrap.cc:492] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.641155 5103 ts_tablet_manager.cc:1367] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.001s
I20240503 02:12:46.641366 5114 ts_tablet_manager.cc:1875] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.642961 4672 raft_consensus.cc:2454] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:46.643478 4462 leader_election.cc:302] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:46.641994 5103 raft_consensus.cc:348] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.644140 6892 raft_consensus.cc:2781] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.644784 6892 raft_consensus.cc:686] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:46.644155 5103 raft_consensus.cc:374] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:12:46.645103 4462 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:46.645171 6892 consensus_queue.cc:237] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.645536 4462 leader_election.cc:302] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:46.646214 6619 raft_consensus.cc:3037] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:46.649096 4462 leader_election.cc:341] T 20ce11c70a924b8d92d8f36fc7401dfd P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:46.650151 6619 raft_consensus.cc:2726] T 1f3dbcbc572f4215b7f59f3750a4b9fa P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.650396 5103 raft_consensus.cc:729] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.650686 5103 consensus_queue.cc:260] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.651332 6814 raft_consensus.cc:482] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.651482 6814 raft_consensus.cc:504] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.652205 5105 tablet_bootstrap.cc:654] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.652204 6814 leader_election.cc:288] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.653791 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5f8d331a718475a9f56dc7f3ec0e6b9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.653859 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5f8d331a718475a9f56dc7f3ec0e6b9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
W20240503 02:12:46.654481 4746 leader_election.cc:341] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:46.654735 4746 leader_election.cc:341] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.654827 4746 leader_election.cc:302] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.655130 5103 ts_tablet_manager.cc:1398] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:46.655457 5103 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.656510 6814 raft_consensus.cc:2726] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.657217 5103 tablet_bootstrap.cc:654] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.662890 5115 ts_tablet_manager.cc:1888] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.667263 5114 ts_tablet_manager.cc:1888] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.667876 5115 log.cc:1196] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/8a90ebcf29004d1a925b69fa20998dd3
I20240503 02:12:46.668498 5115 ts_tablet_manager.cc:1909] T 8a90ebcf29004d1a925b69fa20998dd3 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.669854 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 827) took 5297 ms. Trace:
I20240503 02:12:46.670987 5115 rpcz_store.cc:276] 0503 02:12:41.372689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.372752 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:46.615087 (+5242335us) ts_tablet_manager.cc:1120] Deleting tablet 8a90ebcf29004d1a925b69fa20998dd3
0503 02:12:46.615097 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.649202 (+ 34105us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.662744 (+ 13542us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.669828 (+ 7084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5242128}
I20240503 02:12:46.672127 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet ea5e4db5204e4acf8719100d02d533fb (table default.Table_1d1adcf60a414b9b8d0d7621be5c21c8 [id=d69cde1199bf40368951894b8b8719dd]) successfully deleted
I20240503 02:12:46.673200 5115 tablet_replica.cc:330] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.673707 5115 raft_consensus.cc:2229] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.674458 5115 raft_consensus.cc:2258] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.672886 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043ba14eaa5b46319b0b88457d2a5f45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.674798 4518 raft_consensus.cc:2454] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.675145 5114 log.cc:1196] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:46.675585 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 95681e67db1e4485876d530133e8a514 (table default.Table_2193a952b28f4000b0472d610e708f83 [id=7bb3dfa1e201447cbbe4b81dbfcde0a5]) successfully deleted
I20240503 02:12:46.676123 5114 ts_tablet_manager.cc:1909] T 3f9994ca5aea48d3bc25c097dddba3c3 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.680732 6830 raft_consensus.cc:482] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader acc6d33969e74c209310005ae911c405)
I20240503 02:12:46.680896 6830 raft_consensus.cc:504] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:46.681494 6830 leader_election.cc:288] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers acc6d33969e74c209310005ae911c405 (127.4.17.67:35993), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:46.681905 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94fb9d53044d796e2973143d888fb with delete_type TABLET_DATA_DELETED (replaced by tablet 3e94a9c61cb74cc08ca5867ae85ecdf0) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682113 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed525a61c734ee2b23d5d4f319a7d67 with delete_type TABLET_DATA_DELETED (replaced by tablet 16b3d1e7ce83425db0bd56457d9f2544) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682263 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 59855ab3e42b4e019b769007ccbbc360 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682263 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 1436dd3df7284f9bb379236eb9d76836 with delete_type TABLET_DATA_DELETED (replaced by tablet 17a24b5c83b649e79644b08253d851d6) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682443 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682574 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfe6b2b9cab4517ae7cabec4796c9ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.682685 4644 tablet_service.cc:1467] Processing CreateTablet for tablet 8688fd5cd2bb495eb2558cd247db931d (DEFAULT_TABLE table=default.Table_f028f0dfeea1433ca95764504d64c85e [id=16e41e7059554b6089ce860181ff6ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.682837 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 5984a3757146438d8b06a65f74cd3d6d (DEFAULT_TABLE table=default.Table_d6cf353d37364d2098437daadf566b51 [id=2a57ed99efe34fcb88f092d10589ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.683118 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688fd5cd2bb495eb2558cd247db931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.683696 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 07cca97600374070b117e3ef3869042d (DEFAULT_TABLE table=default.Table_2acba999c8d0432caf7f1cc0f404b8d0 [id=76ee42c3af504bfcbe3d57d3aec1638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.684011 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07cca97600374070b117e3ef3869042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.684391 5159 ts_tablet_manager.cc:1888] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:46.684538 5159 log.cc:1196] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ff8d2c0991e849b5afa35e7230db298d
I20240503 02:12:46.684993 5159 ts_tablet_manager.cc:1909] T ff8d2c0991e849b5afa35e7230db298d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:46.685174 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3a9ee3274d488b9f80abc651bd3a34 (DEFAULT_TABLE table=default.Table_b93e282814624167a59c0ee3582f3b3d [id=aa2c745e913549bcb711565f23daa29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.685508 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3a9ee3274d488b9f80abc651bd3a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.685560 4640 tablet_service.cc:1467] Processing CreateTablet for tablet d757bcb48d1042a78b439a31e5b5c86b (DEFAULT_TABLE table=default.Table_b7257d81b4094fc495ca99d263034757 [id=5d8525a7bebb4ef083530aedae96ad64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.685992 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d757bcb48d1042a78b439a31e5b5c86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.689584 4648 tablet_service.cc:1467] Processing CreateTablet for tablet f951c8a82e9545b09779726e96f2d2ec (DEFAULT_TABLE table=default.Table_5bce1cbd35204706ab1c70da3a37d8dd [id=cc8ee641bf614ce1a3ec28f3bd5ec424]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.690032 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951c8a82e9545b09779726e96f2d2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.693374 4636 tablet_service.cc:1467] Processing CreateTablet for tablet f00221dacba3413a9bfc9134b8b902f7 (DEFAULT_TABLE table=default.Table_c9320e1ce9bc491faf424b7db2850839 [id=b57c1c16607f4f208fb915c06079f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.693763 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00221dacba3413a9bfc9134b8b902f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.695871 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 921) took 4382 ms. Trace:
I20240503 02:12:46.695981 5159 rpcz_store.cc:276] 0503 02:12:42.313819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.314035 (+ 216us) service_pool.cc:225] Handling call
0503 02:12:46.401934 (+4087899us) ts_tablet_manager.cc:1120] Deleting tablet ff8d2c0991e849b5afa35e7230db298d
0503 02:12:46.401945 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.665619 (+263674us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.684244 (+ 18625us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.695844 (+ 11600us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4087705}
I20240503 02:12:46.696493 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ff8d2c0991e849b5afa35e7230db298d (table default.Table_8122f1bfb50244dca4a7836db5ef79bb [id=a04586a0d80d4b0fa88bb0a6dc6b235f]) successfully deleted
I20240503 02:12:46.697069 4645 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b6e303e34a465caffed6c9e1e6bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.697975 4639 tablet_service.cc:1467] Processing CreateTablet for tablet f087a02dd90b4f48833b3e98204cd9d6 (DEFAULT_TABLE table=default.Table_997cf815690744c3a6c9732ff0802a5e [id=8c7c910682db42d1bf8fbb204ec6ca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.698424 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f087a02dd90b4f48833b3e98204cd9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.698367 4645 tablet_service.cc:1467] Processing CreateTablet for tablet b9fd946ea2cb481db7506bdfc834dd4a (DEFAULT_TABLE table=default.Table_3f411ccab7cf48818684560067793bfa [id=89eb739dc9b0477eb4d9c38b876a03b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.698693 4645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fd946ea2cb481db7506bdfc834dd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.698812 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 36c2c3983b0246a38be6d5be78aa3e5f (DEFAULT_TABLE table=default.Table_d2eeb6ba099049a1845ced64b6119f4d [id=0de9ca3926fc478f8f86ca0bd5032d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.699226 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c2c3983b0246a38be6d5be78aa3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.699615 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5984a3757146438d8b06a65f74cd3d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.704536 4638 tablet_service.cc:1467] Processing CreateTablet for tablet f23c97f74e2a4981add1f663b04f40c5 (DEFAULT_TABLE table=default.Table_ca7d5d73ff93496b90bdb5aebca4cde5 [id=3b8ca214ab0a49d4afe2a8443214904b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.705030 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23c97f74e2a4981add1f663b04f40c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.708680 6830 raft_consensus.cc:482] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.708842 6830 raft_consensus.cc:504] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.709524 6830 leader_election.cc:288] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.710094 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:46.713816 4638 tablet_service.cc:1467] Processing CreateTablet for tablet 379b80337ec54489b0d7dda1df420c7f (DEFAULT_TABLE table=default.Table_2c2f293e071545c7ae06e32986fbf729 [id=649a464d2c964714a72528b978f0f9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.714525 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379b80337ec54489b0d7dda1df420c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.721309 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 922) took 4255 ms. Trace:
I20240503 02:12:46.721601 5159 rpcz_store.cc:276] 0503 02:12:42.465516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.465581 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:46.721095 (+4255514us) ts_tablet_manager.cc:1120] Deleting tablet 5a0303f1b1194115a7142a00470457c3
0503 02:12:46.721112 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.721274 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255299}
I20240503 02:12:46.722486 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 3570d8b95daa4a18b2b0a69af627e7a4 (DEFAULT_TABLE table=default.Table_fbe14bb502bb465a95363bfcbbded773 [id=7fba00e5678944789de68d6dac6cbdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.723448 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3570d8b95daa4a18b2b0a69af627e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.724680 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 30b003a2fc5c440aafa64242a3fa1adc (DEFAULT_TABLE table=default.Table_2b1d96d5b6c942f0914e000795c0386f [id=a088ed5d80b845ebbbee25e5a31acbd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:46.725028 4170 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.17.66:35355, user_credentials={real_user=slave}} blocked reactor thread for 62294.7us
I20240503 02:12:46.725672 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b003a2fc5c440aafa64242a3fa1adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.722971 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1649e2c9a64f81a58ecb4df695dc8a (DEFAULT_TABLE table=default.Table_29b8558332544b49aefeb049f9452eb8 [id=ecba57e671f549eda5f0894494f3d04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.726727 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1649e2c9a64f81a58ecb4df695dc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.727613 6899 raft_consensus.cc:482] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.727774 6899 raft_consensus.cc:504] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.728462 6899 leader_election.cc:288] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.731645 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cb6432296c4a45850a96c3651ab7bd with delete_type TABLET_DATA_DELETED (replaced by tablet c70d86c941f2445fbd6ff8b0be7e4043) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.731925 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d2c0991e849b5afa35e7230db298d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.731678 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.722925 5106 tablet_bootstrap.cc:492] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.732388 5106 ts_tablet_manager.cc:1367] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.004s
W20240503 02:12:46.733381 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 5a0303f1b1194115a7142a00470457c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a0303f1b1194115a7142a00470457c3 already in progress: creating tablet
I20240503 02:12:46.733995 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 923) took 4264 ms. Trace:
I20240503 02:12:46.734220 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 043ba14eaa5b46319b0b88457d2a5f45 with delete_type TABLET_DATA_DELETED (replaced by tablet c919bda1eaaa4e52828aac3ef337f3e5) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.734813 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.735236 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet b84856917cb84c2a853c9a8a80246704 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:46.735574 5159 rpcz_store.cc:276] 0503 02:12:42.469859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.469913 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:46.733832 (+4263919us) ts_tablet_manager.cc:1120] Deleting tablet 01c61bf67c8a47fb9ef23bcd6c58c448
0503 02:12:46.733847 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.733975 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4263737}
I20240503 02:12:46.735680 5106 raft_consensus.cc:348] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.715416 6769 raft_consensus.cc:482] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.737630 6769 raft_consensus.cc:504] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.737923 5106 raft_consensus.cc:374] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.737996 5106 raft_consensus.cc:729] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.738298 6769 leader_election.cc:288] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.738351 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 938) took 2576 ms. Trace:
I20240503 02:12:46.736887 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
W20240503 02:12:46.739470 4462 leader_election.cc:341] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: ff8d2c0991e849b5afa35e7230db298d
I20240503 02:12:46.738178 5106 consensus_queue.cc:260] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.737107 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367de4ff0a6f4abfa2924a070f387cbd" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:46.740993 4271 master_service.cc:946] ReplaceTablet: received request to replace tablet 76aef6b021da4f5aad7f523b15d484ec from {username='slave'} at 127.0.0.1:36560
I20240503 02:12:46.741150 4812 raft_consensus.cc:3037] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:46.741366 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c61bf67c8a47fb9ef23bcd6c58c448 already in progress: creating tablet
I20240503 02:12:46.741544 5106 ts_tablet_manager.cc:1398] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:46.741840 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 931) took 3934 ms. Trace:
I20240503 02:12:46.741935 5159 rpcz_store.cc:276] 0503 02:12:42.807645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.819511 (+ 11866us) service_pool.cc:225] Handling call
0503 02:12:46.741661 (+3922150us) ts_tablet_manager.cc:1120] Deleting tablet 9e4dabd6e05948b98f02515023e2aaa1
0503 02:12:46.741675 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.741815 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3894318}
I20240503 02:12:46.738662 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d2c0991e849b5afa35e7230db298d" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:46.742391 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9e4dabd6e05948b98f02515023e2aaa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e4dabd6e05948b98f02515023e2aaa1 already in progress: creating tablet
I20240503 02:12:46.742717 4462 leader_election.cc:302] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:46.742799 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 48e713ee37d042309166b0f118e14986 (table default.Table_fb11bd168d9b4a22940b384400a5c6d5 [id=b4c6a45a47e647b1947d0d7f2ea30afa]) successfully deleted
I20240503 02:12:46.743018 6769 raft_consensus.cc:2726] T ff8d2c0991e849b5afa35e7230db298d P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:46.743491 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0c3b9a888b894897b7cc3cf08402febd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c3b9a888b894897b7cc3cf08402febd already in progress: creating tablet
W20240503 02:12:46.743805 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet efd8e0353b5f4ce484ac81d646b5323d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd8e0353b5f4ce484ac81d646b5323d already in progress: creating tablet
I20240503 02:12:46.743974 4812 raft_consensus.cc:2454] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
W20240503 02:12:46.744096 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 28c1a81589df496b92b980353f00e594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28c1a81589df496b92b980353f00e594 already in progress: creating tablet
W20240503 02:12:46.744378 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 39c48b14bc624c28a871d45e9429ee30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39c48b14bc624c28a871d45e9429ee30 already in progress: creating tablet
W20240503 02:12:46.744987 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5f8656a886564f06931039b516162b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8656a886564f06931039b516162b51 already in progress: creating tablet
I20240503 02:12:46.745827 5105 tablet_bootstrap.cc:492] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.746033 5105 ts_tablet_manager.cc:1367] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20240503 02:12:46.746366 5103 tablet_bootstrap.cc:492] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.746593 5103 ts_tablet_manager.cc:1367] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.000s
I20240503 02:12:46.746795 5105 raft_consensus.cc:348] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.746973 5105 raft_consensus.cc:374] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.747015 5105 raft_consensus.cc:729] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.747243 5105 consensus_queue.cc:260] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.747720 5105 ts_tablet_manager.cc:1398] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:46.747963 5105 tablet_bootstrap.cc:492] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.747884 5103 raft_consensus.cc:348] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.748176 5103 raft_consensus.cc:374] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.748279 5103 raft_consensus.cc:729] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.747881 5115 ts_tablet_manager.cc:1875] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.748620 5103 consensus_queue.cc:260] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.749236 5103 ts_tablet_manager.cc:1398] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:46.749572 5103 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.751195 5103 tablet_bootstrap.cc:654] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:46.754395 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3c421694ce1e424da7953d061a6bca3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c421694ce1e424da7953d061a6bca3e already in progress: creating tablet
I20240503 02:12:46.755686 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 936) took 3751 ms. Trace:
I20240503 02:12:46.758011 4271 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet efa7c0c5fc614c16985008b8fba54bab (table default.Table_637aa20655eb485f921eeebcadeee3eb [id=475bf67c3074494eb43e612a6897f57b]). Expected version 2 got 0
I20240503 02:12:46.758993 4518 raft_consensus.cc:3037] T efa7c0c5fc614c16985008b8fba54bab P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.759668 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 8a90ebcf29004d1a925b69fa20998dd3 (table default.Table_114d4d6efe8947578cff18d0c2f235d2 [id=41995c76abc744c1b75efd4e1a4b4383]) successfully deleted
I20240503 02:12:46.758558 5159 rpcz_store.cc:276] 0503 02:12:43.003865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.003928 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:46.749319 (+3745391us) ts_tablet_manager.cc:1120] Deleting tablet b2560f50415444b8b2e497453a558d3a
0503 02:12:46.749330 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.755662 (+ 6332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3745125}
I20240503 02:12:46.760200 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 938) took 3755 ms. Trace:
I20240503 02:12:46.760293 5159 rpcz_store.cc:276] 0503 02:12:43.004688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.004747 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:46.760061 (+3755314us) ts_tablet_manager.cc:1120] Deleting tablet a6de1b96c15c4b44809a03ef99e077ba
0503 02:12:46.760072 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.760181 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755131}
I20240503 02:12:46.760648 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 937) took 3756 ms. Trace:
I20240503 02:12:46.760713 5159 rpcz_store.cc:276] 0503 02:12:43.004543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.005081 (+ 538us) service_pool.cc:225] Handling call
0503 02:12:46.760540 (+3755459us) ts_tablet_manager.cc:1120] Deleting tablet b5f444b22112407ca02fe86ac50167a9
0503 02:12:46.760550 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.760634 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755295}
I20240503 02:12:46.761042 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 945) took 3676 ms. Trace:
I20240503 02:12:46.761107 5159 rpcz_store.cc:276] 0503 02:12:43.085033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.085084 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:46.760936 (+3675852us) ts_tablet_manager.cc:1120] Deleting tablet 192460518d49417cb8f03ac6ec971eef
0503 02:12:46.760945 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.761027 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675660}
I20240503 02:12:46.761328 5159 tablet_replica.cc:330] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:46.761523 5159 raft_consensus.cc:2229] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.762418 5106 tablet_bootstrap.cc:492] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:46.763386 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b2560f50415444b8b2e497453a558d3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2560f50415444b8b2e497453a558d3a already in progress: creating tablet
W20240503 02:12:46.763664 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet a6de1b96c15c4b44809a03ef99e077ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6de1b96c15c4b44809a03ef99e077ba already in progress: creating tablet
W20240503 02:12:46.763837 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet b5f444b22112407ca02fe86ac50167a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f444b22112407ca02fe86ac50167a9 already in progress: creating tablet
W20240503 02:12:46.764001 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 192460518d49417cb8f03ac6ec971eef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 192460518d49417cb8f03ac6ec971eef already in progress: creating tablet
I20240503 02:12:46.738436 5114 rpcz_store.cc:276] 0503 02:12:44.161629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.161681 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:46.637993 (+2476312us) ts_tablet_manager.cc:1120] Deleting tablet 3f9994ca5aea48d3bc25c097dddba3c3
0503 02:12:46.638004 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.663345 (+ 25341us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.667141 (+ 3796us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.738328 (+ 71187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2468265}
I20240503 02:12:46.765256 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 939) took 2603 ms. Trace:
I20240503 02:12:46.765242 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 3f9994ca5aea48d3bc25c097dddba3c3 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
I20240503 02:12:46.765345 5114 rpcz_store.cc:276] 0503 02:12:44.161857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.161905 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:46.765129 (+2603224us) ts_tablet_manager.cc:1120] Deleting tablet 73965590a0ea4bffae6e99fa1367fad0
0503 02:12:46.765142 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.765237 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595247}
I20240503 02:12:46.765746 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 940) took 2567 ms. Trace:
W20240503 02:12:46.765795 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 73965590a0ea4bffae6e99fa1367fad0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73965590a0ea4bffae6e99fa1367fad0 already in progress: creating tablet
I20240503 02:12:46.765856 5114 rpcz_store.cc:276] 0503 02:12:44.198762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.198830 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:46.765641 (+2566811us) ts_tablet_manager.cc:1120] Deleting tablet b9470480a9de45209044f8e8c461519a
0503 02:12:46.765650 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.765731 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2566615}
I20240503 02:12:46.766254 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 941) took 2565 ms. Trace:
I20240503 02:12:46.766325 5114 rpcz_store.cc:276] 0503 02:12:44.201148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.201203 (+ 55us) service_pool.cc:225] Handling call
0503 02:12:46.766105 (+2564902us) ts_tablet_manager.cc:1120] Deleting tablet 8d1ac81e1a6f41cf974bb30e6b74eea2
0503 02:12:46.766114 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.766238 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2564706}
W20240503 02:12:46.766669 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 already in progress: creating tablet
I20240503 02:12:46.766906 5106 tablet_bootstrap.cc:654] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.767009 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 942) took 2562 ms. Trace:
I20240503 02:12:46.767071 5114 rpcz_store.cc:276] 0503 02:12:44.205013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.205067 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:46.766905 (+2561838us) ts_tablet_manager.cc:1120] Deleting tablet 9d900a0dc6c34076a0993210693a57a0
0503 02:12:46.766914 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.766992 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2561645}
W20240503 02:12:46.767458 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9d900a0dc6c34076a0993210693a57a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d900a0dc6c34076a0993210693a57a0 already in progress: creating tablet
I20240503 02:12:46.767769 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 943) took 2558 ms. Trace:
I20240503 02:12:46.767833 5114 rpcz_store.cc:276] 0503 02:12:44.208943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.209080 (+ 137us) service_pool.cc:225] Handling call
0503 02:12:46.767675 (+2558595us) ts_tablet_manager.cc:1120] Deleting tablet 7c19263f06ec41d7a6fe6a8df7f88f8f
0503 02:12:46.767682 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.767754 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2558356}
I20240503 02:12:46.768225 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 944) took 2557 ms. Trace:
I20240503 02:12:46.768299 5114 rpcz_store.cc:276] 0503 02:12:44.210607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.210659 (+ 52us) service_pool.cc:225] Handling call
0503 02:12:46.768123 (+2557464us) ts_tablet_manager.cc:1120] Deleting tablet 89f3232d509f4cb39eb8923fe1d120a7
0503 02:12:46.768133 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.768209 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2557266}
W20240503 02:12:46.768235 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c19263f06ec41d7a6fe6a8df7f88f8f already in progress: creating tablet
W20240503 02:12:46.768811 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 89f3232d509f4cb39eb8923fe1d120a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f3232d509f4cb39eb8923fe1d120a7 already in progress: creating tablet
I20240503 02:12:46.768843 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 946) took 2375 ms. Trace:
I20240503 02:12:46.769037 5114 rpcz_store.cc:276] 0503 02:12:44.393919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.394104 (+ 185us) service_pool.cc:225] Handling call
0503 02:12:46.768743 (+2374639us) ts_tablet_manager.cc:1120] Deleting tablet 1b13ea83ffa54dcbbfebaaa76e1686a9
0503 02:12:46.768754 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.768828 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2374431}
I20240503 02:12:46.770367 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.770601 4812 raft_consensus.cc:2454] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.773442 5159 raft_consensus.cc:2258] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.774474 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.774685 4812 raft_consensus.cc:2454] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:46.776084 4608 consensus_peers.cc:485] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b8db679ba24ac6a93be591c0513ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:46.777168 5159 ts_tablet_manager.cc:1875] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:46.778589 4608 consensus_peers.cc:485] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:46.780733 5105 tablet_bootstrap.cc:654] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.782682 4608 leader_election.cc:302] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:46.785383 6901 raft_consensus.cc:2781] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.785511 6901 raft_consensus.cc:482] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.785566 6901 raft_consensus.cc:3037] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.788581 6901 raft_consensus.cc:504] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.789857 6901 leader_election.cc:288] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:46.790555 4608 consensus_peers.cc:485] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:46.791249 4608 leader_election.cc:341] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.792600 5115 ts_tablet_manager.cc:1888] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.793664 5105 tablet_bootstrap.cc:492] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.793895 5105 ts_tablet_manager.cc:1367] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240503 02:12:46.795100 5105 raft_consensus.cc:348] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.795466 5105 raft_consensus.cc:374] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.795589 5105 raft_consensus.cc:729] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.795814 6901 consensus_queue.cc:1035] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.796123 5115 log.cc:1196] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b9511e4de6c942b9b9e0790c88d3d543
I20240503 02:12:46.796005 5105 consensus_queue.cc:260] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:46.797391 4608 consensus_peers.cc:485] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd66f88a24b74f2c9b2050af081d4601. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:46.797860 5105 ts_tablet_manager.cc:1398] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240503 02:12:46.798568 4608 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Not found: Tablet not found: c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:46.799033 5105 tablet_bootstrap.cc:492] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.799367 6901 consensus_queue.cc:1035] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.799902 5115 ts_tablet_manager.cc:1909] T b9511e4de6c942b9b9e0790c88d3d543 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.800379 6774 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:12:46.801452 6814 raft_consensus.cc:482] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.801595 6814 raft_consensus.cc:504] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.802218 6814 leader_election.cc:288] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.803244 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a1d5b2de8d4d3ba5190750545c9f7b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:46.803844 4746 leader_election.cc:341] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.769673 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.774322 4518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.807214 4525 raft_consensus.cc:2454] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.803808 6769 raft_consensus.cc:482] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.807462 6769 raft_consensus.cc:504] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:46.807636 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 already in progress: creating tablet
I20240503 02:12:46.807981 5114 tablet_replica.cc:330] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.808113 6769 leader_election.cc:288] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.808157 5114 raft_consensus.cc:2229] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.808527 5114 raft_consensus.cc:2258] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.804246 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a1d5b2de8d4d3ba5190750545c9f7b" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.809125 5114 ts_tablet_manager.cc:1875] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:46.809334 4746 leader_election.cc:341] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.809477 4746 leader_election.cc:302] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:46.809947 4462 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 1436dd3df7284f9bb379236eb9d76836
I20240503 02:12:46.810209 6814 raft_consensus.cc:2726] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.811848 5105 tablet_bootstrap.cc:654] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.812546 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1436dd3df7284f9bb379236eb9d76836" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.812772 4667 raft_consensus.cc:2454] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 0.
I20240503 02:12:46.813020 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 850) took 5090 ms. Trace:
I20240503 02:12:46.813370 4462 leader_election.cc:302] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:46.813674 6769 raft_consensus.cc:2781] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.813751 6769 raft_consensus.cc:482] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.813820 6769 raft_consensus.cc:3037] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.816450 6769 raft_consensus.cc:504] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.818987 4165 master-stress-test.cc:580] Killed test thread 21/25
I20240503 02:12:46.819096 4165 master-stress-test.cc:578] Killing test thread 22/25
I20240503 02:12:46.819169 5103 tablet_bootstrap.cc:492] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:46.819741 6769 leader_election.cc:288] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.816884 5159 ts_tablet_manager.cc:1888] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 02:12:46.821213 4462 leader_election.cc:341] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 1436dd3df7284f9bb379236eb9d76836
I20240503 02:12:46.819314 5103 ts_tablet_manager.cc:1367] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20240503 02:12:46.822441 5103 raft_consensus.cc:348] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.822762 5103 raft_consensus.cc:374] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.814059 5106 tablet_bootstrap.cc:492] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.822886 5103 raft_consensus.cc:729] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.815057 6901 consensus_queue.cc:1035] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:46.815771 4608 leader_election.cc:341] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 043ba14eaa5b46319b0b88457d2a5f45
I20240503 02:12:46.820333 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1436dd3df7284f9bb379236eb9d76836" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:46.823171 4667 raft_consensus.cc:3037] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.823367 5106 ts_tablet_manager.cc:1367] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.004s
I20240503 02:12:46.824077 5106 raft_consensus.cc:348] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.824291 5106 raft_consensus.cc:374] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.824352 5106 raft_consensus.cc:729] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.824610 5106 consensus_queue.cc:260] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.825341 5106 ts_tablet_manager.cc:1398] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:46.825644 5106 tablet_bootstrap.cc:492] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.826141 4667 raft_consensus.cc:2454] T 1436dd3df7284f9bb379236eb9d76836 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9f079230a468f90e5e3a58f254b91 in term 1.
I20240503 02:12:46.827071 5106 tablet_bootstrap.cc:654] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.827666 4608 leader_election.cc:302] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.827745 5115 rpcz_store.cc:276] 0503 02:12:41.722928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.723057 (+ 129us) service_pool.cc:225] Handling call
0503 02:12:46.673124 (+4950067us) ts_tablet_manager.cc:1120] Deleting tablet b9511e4de6c942b9b9e0790c88d3d543
0503 02:12:46.673137 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.777456 (+104319us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.792476 (+ 15020us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.812994 (+ 20518us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3764,"tablet-delete.queue_time_us":4949841}
I20240503 02:12:46.827991 6901 raft_consensus.cc:2781] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.828178 6901 raft_consensus.cc:686] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:46.828454 6901 consensus_queue.cc:237] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.829152 5115 tablet_replica.cc:330] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.829438 5115 raft_consensus.cc:2229] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.829818 5115 raft_consensus.cc:2258] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.830495 5115 ts_tablet_manager.cc:1875] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:46.828438 4608 leader_election.cc:341] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:46.832306 4608 leader_election.cc:302] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81; no voters: acc6d33969e74c209310005ae911c405, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.833019 6899 raft_consensus.cc:2726] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 02:12:46.833683 4608 leader_election.cc:302] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:46.834820 6830 raft_consensus.cc:2781] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.834882 4608 leader_election.cc:302] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:46.826970 4462 leader_election.cc:302] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters: acc6d33969e74c209310005ae911c405
I20240503 02:12:46.835441 6769 raft_consensus.cc:2781] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.835574 6769 raft_consensus.cc:686] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Becoming Leader. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Running, Role: LEADER
I20240503 02:12:46.835892 6769 consensus_queue.cc:237] T 1436dd3df7284f9bb379236eb9d76836 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.836251 5103 consensus_queue.cc:260] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.834900 6830 raft_consensus.cc:482] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.836807 6830 raft_consensus.cc:3037] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.837007 6899 raft_consensus.cc:2781] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.837116 6899 raft_consensus.cc:482] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.837214 6899 raft_consensus.cc:3037] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.839623 6830 raft_consensus.cc:504] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.840162 6899 raft_consensus.cc:504] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.840284 6830 leader_election.cc:288] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.840766 6899 leader_election.cc:288] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.842274 6899 consensus_queue.cc:1035] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:12:46.845239 5103 ts_tablet_manager.cc:1398] T 367de4ff0a6f4abfa2924a070f387cbd P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240503 02:12:46.846549 5106 tablet_bootstrap.cc:492] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.846767 5106 ts_tablet_manager.cc:1367] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20240503 02:12:46.847505 5103 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:46.847573 5106 raft_consensus.cc:348] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.847934 5106 raft_consensus.cc:374] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.848060 5106 raft_consensus.cc:729] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.848227 5159 log.cc:1196] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/84b6e303e34a465caffed6c9e1e6bf74
I20240503 02:12:46.848395 5106 consensus_queue.cc:260] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.849059 5106 ts_tablet_manager.cc:1398] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:46.849649 5106 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.849761 5159 ts_tablet_manager.cc:1909] T 84b6e303e34a465caffed6c9e1e6bf74 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:46.848528 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043ba14eaa5b46319b0b88457d2a5f45" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:46.851128 5106 tablet_bootstrap.cc:654] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.851217 4527 raft_consensus.cc:3037] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.851917 5103 tablet_bootstrap.cc:654] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.854545 4527 raft_consensus.cc:2454] T 043ba14eaa5b46319b0b88457d2a5f45 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:46.856736 5114 ts_tablet_manager.cc:1888] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.856966 5114 log.cc:1196] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/607e49cde2fd4aab9f44921d3c3921a5
I20240503 02:12:46.857537 5114 ts_tablet_manager.cc:1909] T 607e49cde2fd4aab9f44921d3c3921a5 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:46.861847 5115 ts_tablet_manager.cc:1888] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:46.862006 5115 log.cc:1196] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/66ca08877cd948e1b04736b54bc779db
I20240503 02:12:46.862668 5115 ts_tablet_manager.cc:1909] T 66ca08877cd948e1b04736b54bc779db P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.863523 6899 raft_consensus.cc:482] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.863651 6899 raft_consensus.cc:504] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.864238 6899 leader_election.cc:288] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.861907 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 945) took 2547 ms. Trace:
I20240503 02:12:46.865347 5114 rpcz_store.cc:276] 0503 02:12:44.313984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.314047 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:46.807920 (+2493873us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:46.807933 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.829579 (+ 21646us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.856605 (+ 27026us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.861879 (+ 5274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349804}
I20240503 02:12:46.865967 5106 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:46.866055 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 607e49cde2fd4aab9f44921d3c3921a5 (table default.Table_0e5ff92986ff447dbc2c9e69a628e8fe [id=08bfe029472248d7b17aaa9a8ee63472]) successfully deleted
I20240503 02:12:46.866166 5106 ts_tablet_manager.cc:1367] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 02:12:46.866621 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 951) took 2383 ms. Trace:
I20240503 02:12:46.866832 5106 raft_consensus.cc:348] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.866925 5114 rpcz_store.cc:276] 0503 02:12:44.482735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.482797 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:46.866481 (+2383684us) ts_tablet_manager.cc:1120] Deleting tablet 7219ce14a9ae4488b97914b9d02995f9
0503 02:12:46.866492 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.866602 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2383449}
I20240503 02:12:46.867071 5106 raft_consensus.cc:374] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.867134 5106 raft_consensus.cc:729] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
W20240503 02:12:46.867511 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7219ce14a9ae4488b97914b9d02995f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7219ce14a9ae4488b97914b9d02995f9 already in progress: creating tablet
I20240503 02:12:46.867437 5106 consensus_queue.cc:260] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.867949 5114 tablet_replica.cc:330] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:46.868180 5114 raft_consensus.cc:2229] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.868203 5106 ts_tablet_manager.cc:1398] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:46.868432 5114 raft_consensus.cc:2258] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.868475 5106 tablet_bootstrap.cc:492] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:46.868800 4519 raft_consensus.cc:3037] T c2efdbfa909c475d95e99cf24206608f P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.869226 5114 ts_tablet_manager.cc:1875] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.869940 5106 tablet_bootstrap.cc:654] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.870982 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 851) took 5147 ms. Trace:
I20240503 02:12:46.871074 5115 rpcz_store.cc:276] 0503 02:12:41.723018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.733273 (+ 10255us) service_pool.cc:225] Handling call
0503 02:12:46.829094 (+5095821us) ts_tablet_manager.cc:1120] Deleting tablet 66ca08877cd948e1b04736b54bc779db
0503 02:12:46.829106 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.845089 (+ 15983us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.861720 (+ 16631us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.870960 (+ 9240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5095571}
I20240503 02:12:46.871408 5115 tablet_replica.cc:330] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.871610 5115 raft_consensus.cc:2229] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.871798 5115 raft_consensus.cc:2258] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.872354 5115 ts_tablet_manager.cc:1875] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:46.880483 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
I20240503 02:12:46.880753 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (table default.Table_ec81cfea38bf4719b2e30c82d4d93f50 [id=997231149d3e427f824ed751b345d7b8]). Expected version 2 got 0
I20240503 02:12:46.882588 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b9511e4de6c942b9b9e0790c88d3d543 (table default.Table_b417b8282cd14a9aa9bcffd17aab880c [id=e22df2e2673c45cbb1b32201a728333c]) successfully deleted
I20240503 02:12:46.886255 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:46.886605 4527 raft_consensus.cc:3037] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.887044 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:46.888399 5115 ts_tablet_manager.cc:1888] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240503 02:12:46.888542 5115 log.cc:1196] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c88eaba979394b449d96ee956c3f28a8
I20240503 02:12:46.889019 5115 ts_tablet_manager.cc:1909] T c88eaba979394b449d96ee956c3f28a8 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:46.883073 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:46.890029 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 853) took 5048 ms. Trace:
I20240503 02:12:46.866497 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 946) took 3780 ms. Trace:
I20240503 02:12:46.890255 5159 rpcz_store.cc:276] 0503 02:12:43.085548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.085596 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:46.761284 (+3675688us) ts_tablet_manager.cc:1120] Deleting tablet 84b6e303e34a465caffed6c9e1e6bf74
0503 02:12:46.761292 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.794570 (+ 33278us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.816762 (+ 22192us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.866471 (+ 49709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675513}
I20240503 02:12:46.890779 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 84b6e303e34a465caffed6c9e1e6bf74 (table default.Table_634134e8f4a14d2a86f04be0d642a853 [id=e6681113e47842469aada48896063a12]) successfully deleted
I20240503 02:12:46.891098 6814 raft_consensus.cc:482] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.891238 6814 raft_consensus.cc:504] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.891914 6814 leader_election.cc:288] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.892800 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 947) took 3508 ms. Trace:
I20240503 02:12:46.893136 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb174825f1441f861ae6816137f855" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.893283 5159 rpcz_store.cc:276] 0503 02:12:43.382059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.382206 (+ 147us) service_pool.cc:225] Handling call
0503 02:12:46.890749 (+3508543us) ts_tablet_manager.cc:1120] Deleting tablet cbaa94834cdc44eabe71242a698e3639
0503 02:12:46.890762 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.890894 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3508330}
I20240503 02:12:46.894662 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 948) took 3512 ms. Trace:
I20240503 02:12:46.895372 4521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb174825f1441f861ae6816137f855" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.895530 5159 rpcz_store.cc:276] 0503 02:12:43.382187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.382786 (+ 599us) service_pool.cc:225] Handling call
0503 02:12:46.894513 (+3511727us) ts_tablet_manager.cc:1120] Deleting tablet f8be9e5380b4466baccd16483e940588
0503 02:12:46.894528 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.894640 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3511550}
W20240503 02:12:46.896024 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet f8be9e5380b4466baccd16483e940588 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8be9e5380b4466baccd16483e940588 already in progress: creating tablet
W20240503 02:12:46.896473 4746 leader_election.cc:341] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:46.896670 4746 leader_election.cc:341] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.896734 4746 leader_election.cc:302] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.897424 6814 raft_consensus.cc:2726] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.898453 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1fac803dd84eee8722f0a14559faf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:46.898660 4608 leader_election.cc:341] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.899552 4608 leader_election.cc:302] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91; no voters:
I20240503 02:12:46.885886 5105 tablet_bootstrap.cc:492] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:46.901832 5105 ts_tablet_manager.cc:1367] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.000s
I20240503 02:12:46.886654 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c099a306a56643069f63e34a5f0d3dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:46.902503 5105 raft_consensus.cc:348] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.902702 5105 raft_consensus.cc:374] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.902748 5105 raft_consensus.cc:729] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.887334 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec641e403ed843fda31dabd3920aeb68" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:46.902834 4809 raft_consensus.cc:3037] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.902972 5105 consensus_queue.cc:260] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.903645 5105 ts_tablet_manager.cc:1398] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:46.903942 5105 tablet_bootstrap.cc:492] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:46.902604 4813 raft_consensus.cc:3037] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.908609 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 087fc9564e2f405b82069448757a2d8e (DEFAULT_TABLE table=default.Table_370e31b9745e49a69e323d187f951fc9 [id=38af125dc8b14378a4987f82a424b5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.909184 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087fc9564e2f405b82069448757a2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.910503 4813 raft_consensus.cc:2454] T c099a306a56643069f63e34a5f0d3dd7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:46.911516 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 087fc9564e2f405b82069448757a2d8e (DEFAULT_TABLE table=default.Table_370e31b9745e49a69e323d187f951fc9 [id=38af125dc8b14378a4987f82a424b5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.911731 4809 raft_consensus.cc:2454] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:46.911984 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087fc9564e2f405b82069448757a2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.912521 6814 raft_consensus.cc:482] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.912680 6814 raft_consensus.cc:504] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.914605 4500 tablet_service.cc:1467] Processing CreateTablet for tablet c1eb672d91914513a4b04a9a5c70726a (DEFAULT_TABLE table=default.Table_62f0644508ee4e159ae0fd95cb3558c2 [id=f38b3ca16f5b4deb99967782dd73dfdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.915096 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eb672d91914513a4b04a9a5c70726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.916285 6892 raft_consensus.cc:482] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.916497 6892 raft_consensus.cc:504] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.917282 6892 leader_election.cc:288] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.917325 4493 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6dd1cc17942139bda7d85c875e16b (DEFAULT_TABLE table=default.Table_dbb9132d9f3441548f0e10b9818769f8 [id=54353980644249588665c95a13c842d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.917820 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6dd1cc17942139bda7d85c875e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.890173 5115 rpcz_store.cc:276] 0503 02:12:41.841258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.841311 (+ 53us) service_pool.cc:225] Handling call
0503 02:12:46.871358 (+5030047us) ts_tablet_manager.cc:1120] Deleting tablet c88eaba979394b449d96ee956c3f28a8
0503 02:12:46.871368 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.880769 (+ 9401us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.888288 (+ 7519us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:46.889994 (+ 1706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5029854}
I20240503 02:12:46.918646 4791 tablet_service.cc:1467] Processing CreateTablet for tablet c1eb672d91914513a4b04a9a5c70726a (DEFAULT_TABLE table=default.Table_62f0644508ee4e159ae0fd95cb3558c2 [id=f38b3ca16f5b4deb99967782dd73dfdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.919087 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eb672d91914513a4b04a9a5c70726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.919464 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 852) took 5081 ms. Trace:
I20240503 02:12:46.919554 5115 rpcz_store.cc:276] 0503 02:12:41.838016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.841942 (+ 3926us) service_pool.cc:225] Handling call
0503 02:12:46.919300 (+5077358us) ts_tablet_manager.cc:1120] Deleting tablet 70e08c10c15f41878202cef0c8d5ad3c
0503 02:12:46.919312 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.919440 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077193}
I20240503 02:12:46.919927 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 854) took 5076 ms. Trace:
I20240503 02:12:46.920009 5115 rpcz_store.cc:276] 0503 02:12:41.843914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.847259 (+ 3345us) service_pool.cc:225] Handling call
0503 02:12:46.919806 (+5072547us) ts_tablet_manager.cc:1120] Deleting tablet 7b86a6bb791449b5a4c490663d87afd4
0503 02:12:46.919813 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.919910 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5072320}
I20240503 02:12:46.920310 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 855) took 5064 ms. Trace:
I20240503 02:12:46.920372 5115 rpcz_store.cc:276] 0503 02:12:41.855544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.855609 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:46.920216 (+5064607us) ts_tablet_manager.cc:1120] Deleting tablet 7982ba6234b844f682cb23c0bfc4068c
0503 02:12:46.920224 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.920296 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5064268}
I20240503 02:12:46.920624 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 856) took 5060 ms. Trace:
I20240503 02:12:46.920677 5115 rpcz_store.cc:276] 0503 02:12:41.860015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:41.860115 (+ 100us) service_pool.cc:225] Handling call
0503 02:12:46.920541 (+5060426us) ts_tablet_manager.cc:1120] Deleting tablet ee16f1beb6a34a85bc872dedd64472ce
0503 02:12:46.920548 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.920612 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5060066}
I20240503 02:12:46.920964 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 868) took 4703 ms. Trace:
I20240503 02:12:46.921030 5115 rpcz_store.cc:276] 0503 02:12:42.217112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.218295 (+ 1183us) service_pool.cc:225] Handling call
0503 02:12:46.920841 (+4702546us) ts_tablet_manager.cc:1120] Deleting tablet 1a082f4c43fd4c188be9ee3966e26edf
0503 02:12:46.920848 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.920949 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4702372}
I20240503 02:12:46.921300 5115 tablet_replica.cc:330] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:46.921489 5115 raft_consensus.cc:2229] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.921811 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed525a61c734ee2b23d5d4f319a7d67" candidate_uuid: "f4f9f079230a468f90e5e3a58f254b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.922107 4667 raft_consensus.cc:2379] T 9ed525a61c734ee2b23d5d4f319a7d67 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4f9f079230a468f90e5e3a58f254b91 in current term 1: Already voted for candidate 1f50719d94564844b15e1a9211d48a81 in this term.
I20240503 02:12:46.923321 6814 leader_election.cc:288] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:46.926927 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d5eac44264899a7a372b9f85fb8f5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:46.927367 4267 master_service.cc:946] ReplaceTablet: received request to replace tablet c88eaba979394b449d96ee956c3f28a8 from {username='slave'} at 127.0.0.1:36560
W20240503 02:12:46.928201 4746 leader_election.cc:341] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.928740 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d5eac44264899a7a372b9f85fb8f5" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:46.937584 5105 tablet_bootstrap.cc:654] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.941797 4499 tablet_service.cc:1467] Processing CreateTablet for tablet bea254eeb9b84fc283032704fb450d7f (DEFAULT_TABLE table=default.Table_bea552b4d4954b9b80d91b223a4349ce [id=1cb7f252528a439db33f7877dc8f2771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.942029 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 68a60272d2e14853a6f5e6c7d07f8b74 (DEFAULT_TABLE table=default.Table_6d1f952d30ef4ea4ba59b6b7ebf7548d [id=e1031d4cc15c4aacbe97975af2991ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.942325 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea254eeb9b84fc283032704fb450d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.942492 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a60272d2e14853a6f5e6c7d07f8b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.944545 4491 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3229f7010a4d458e8d7ee82f7df499 (DEFAULT_TABLE table=default.Table_2bc261e8aa5a478786fbc5296356f0d5 [id=c5dae91c96ca47f4b7aff913fbcb5494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.944981 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3229f7010a4d458e8d7ee82f7df499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.945190 4505 tablet_service.cc:1467] Processing CreateTablet for tablet b7a366b6b96346e9b9dc454613db662e (DEFAULT_TABLE table=default.Table_6dd79385e5714ac1828bcf7463efbd5e [id=d8980a9ef229444d8f888769979b0f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.945576 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a366b6b96346e9b9dc454613db662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.947082 4506 tablet_service.cc:1467] Processing CreateTablet for tablet dbd23f93e788468b99dba3ffa1398450 (DEFAULT_TABLE table=default.Table_0f2571bf88ac4a23b6b806c2ae05671b [id=8eefd98351bc4f1ca157b595ddda25a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.947449 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd23f93e788468b99dba3ffa1398450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.945200 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d (DEFAULT_TABLE table=default.Table_552f6a00a910447e9ee59068d1d3908f [id=bf6c7fd1504f4ed8a53148da7e7806f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.949455 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.941814 4509 tablet_service.cc:1467] Processing CreateTablet for tablet d96c2ce647614957804fc5689a0ee0a4 (DEFAULT_TABLE table=default.Table_e12eac6a8aaa48a9b27f9995e7e36b10 [id=cc88cd0e88ed4908869660446ffa7df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.953387 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96c2ce647614957804fc5689a0ee0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.954340 6830 raft_consensus.cc:2781] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.954490 6830 raft_consensus.cc:686] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:46.954741 6830 consensus_queue.cc:237] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.959753 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 54959b3e30944fab8bd5fd47ce42d246 (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]). Expected version 2 got 0
I20240503 02:12:46.960084 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 442d795ee01c417dbdeda41fd6853dd7 (table default.Table_512dd0dd609548978d52d05cc8e69782 [id=52915748a54142109318ab8a12dea14f]). Expected version 1 got 0
I20240503 02:12:46.960451 5115 raft_consensus.cc:2258] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.961539 5115 ts_tablet_manager.cc:1875] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:46.963037 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 63486.7us
W20240503 02:12:46.963483 4608 consensus_peers.cc:485] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:46.963712 4608 leader_election.cc:341] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.964586 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
W20240503 02:12:46.937857 4746 leader_election.cc:341] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:46.942332 4783 tablet_service.cc:1467] Processing CreateTablet for tablet bea254eeb9b84fc283032704fb450d7f (DEFAULT_TABLE table=default.Table_bea552b4d4954b9b80d91b223a4349ce [id=1cb7f252528a439db33f7877dc8f2771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.977620 4746 leader_election.cc:302] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:46.977988 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea254eeb9b84fc283032704fb450d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.978122 6941 raft_consensus.cc:2726] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:46.978706 6841 raft_consensus.cc:482] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:46.978868 6841 raft_consensus.cc:504] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:46.979560 6841 leader_election.cc:288] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:46.942740 4776 tablet_service.cc:1467] Processing CreateTablet for tablet b7a366b6b96346e9b9dc454613db662e (DEFAULT_TABLE table=default.Table_6dd79385e5714ac1828bcf7463efbd5e [id=d8980a9ef229444d8f888769979b0f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.981292 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a366b6b96346e9b9dc454613db662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.943117 4781 tablet_service.cc:1467] Processing CreateTablet for tablet d96c2ce647614957804fc5689a0ee0a4 (DEFAULT_TABLE table=default.Table_e12eac6a8aaa48a9b27f9995e7e36b10 [id=cc88cd0e88ed4908869660446ffa7df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.985308 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96c2ce647614957804fc5689a0ee0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.986692 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a1d5b2de8d4d3ba5190750545c9f7b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:46.986917 4809 raft_consensus.cc:2454] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:46.987370 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 66ca08877cd948e1b04736b54bc779db (table default.Table_ec81cfea38bf4719b2e30c82d4d93f50 [id=997231149d3e427f824ed751b345d7b8]) successfully deleted
I20240503 02:12:46.943401 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 68a60272d2e14853a6f5e6c7d07f8b74 (DEFAULT_TABLE table=default.Table_6d1f952d30ef4ea4ba59b6b7ebf7548d [id=e1031d4cc15c4aacbe97975af2991ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.948103 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3229f7010a4d458e8d7ee82f7df499 (DEFAULT_TABLE table=default.Table_2bc261e8aa5a478786fbc5296356f0d5 [id=c5dae91c96ca47f4b7aff913fbcb5494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.989269 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a60272d2e14853a6f5e6c7d07f8b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.989459 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3229f7010a4d458e8d7ee82f7df499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.952497 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d (DEFAULT_TABLE table=default.Table_552f6a00a910447e9ee59068d1d3908f [id=bf6c7fd1504f4ed8a53148da7e7806f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.992786 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.952968 4778 tablet_service.cc:1467] Processing CreateTablet for tablet dbd23f93e788468b99dba3ffa1398450 (DEFAULT_TABLE table=default.Table_0f2571bf88ac4a23b6b806c2ae05671b [id=8eefd98351bc4f1ca157b595ddda25a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.996973 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd23f93e788468b99dba3ffa1398450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.998088 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c88eaba979394b449d96ee956c3f28a8 (table default.Table_e66e7c44c2784f97bc80c180112e3e11 [id=62c39d7fbcb3410994f6599d86741094]) successfully deleted
W20240503 02:12:46.999047 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 70e08c10c15f41878202cef0c8d5ad3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e08c10c15f41878202cef0c8d5ad3c already in progress: creating tablet
I20240503 02:12:46.953248 4791 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6dd1cc17942139bda7d85c875e16b (DEFAULT_TABLE table=default.Table_dbb9132d9f3441548f0e10b9818769f8 [id=54353980644249588665c95a13c842d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.001363 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6dd1cc17942139bda7d85c875e16b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:47.004123 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7b86a6bb791449b5a4c490663d87afd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b86a6bb791449b5a4c490663d87afd4 already in progress: creating tablet
W20240503 02:12:47.004695 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7982ba6234b844f682cb23c0bfc4068c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7982ba6234b844f682cb23c0bfc4068c already in progress: creating tablet
I20240503 02:12:47.005040 5115 ts_tablet_manager.cc:1888] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.005213 5115 log.cc:1196] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/42e80cc31f5c4908b82f2b030255f6af
I20240503 02:12:47.005581 5115 ts_tablet_manager.cc:1909] T 42e80cc31f5c4908b82f2b030255f6af P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:47.005556 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ee16f1beb6a34a85bc872dedd64472ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee16f1beb6a34a85bc872dedd64472ce already in progress: creating tablet
W20240503 02:12:47.005793 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1a082f4c43fd4c188be9ee3966e26edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a082f4c43fd4c188be9ee3966e26edf already in progress: creating tablet
I20240503 02:12:47.006778 4812 raft_consensus.cc:1261] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Refusing update from remote peer 1f50719d94564844b15e1a9211d48a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:47.007277 6901 raft_consensus.cc:482] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:47.007457 6901 raft_consensus.cc:504] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.008215 6830 raft_consensus.cc:482] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:47.008409 6830 raft_consensus.cc:504] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:47.009112 4462 leader_election.cc:341] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 9ed525a61c734ee2b23d5d4f319a7d67
I20240503 02:12:47.009284 4462 leader_election.cc:302] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4f9f079230a468f90e5e3a58f254b91; no voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405
I20240503 02:12:47.009629 6830 leader_election.cc:288] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.009896 6892 raft_consensus.cc:3037] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.013510 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a1d5b2de8d4d3ba5190750545c9f7b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:47.013919 6892 raft_consensus.cc:2726] T 9ed525a61c734ee2b23d5d4f319a7d67 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:47.015234 6901 leader_election.cc:288] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.019405 4165 master-stress-test.cc:580] Killed test thread 22/25
I20240503 02:12:47.019613 4165 master-stress-test.cc:578] Killing test thread 23/25
I20240503 02:12:47.019706 4165 master-stress-test.cc:580] Killed test thread 23/25
I20240503 02:12:47.019757 4165 master-stress-test.cc:578] Killing test thread 24/25
I20240503 02:12:47.019827 4165 master-stress-test.cc:580] Killed test thread 24/25
I20240503 02:12:47.019874 4165 master-stress-test.cc:578] Killing test thread 25/25
I20240503 02:12:47.019949 4165 master-stress-test.cc:580] Killed test thread 25/25
I20240503 02:12:47.020098 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4423
I20240503 02:12:47.022102 5106 tablet_bootstrap.cc:492] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.022444 5106 ts_tablet_manager.cc:1367] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.014s
I20240503 02:12:47.023563 5106 raft_consensus.cc:348] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.024055 5106 raft_consensus.cc:374] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.024184 5106 raft_consensus.cc:729] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.024565 5106 consensus_queue.cc:260] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.025920 5106 ts_tablet_manager.cc:1398] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:47.027412 5106 tablet_bootstrap.cc:492] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.029114 5103 tablet_bootstrap.cc:492] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:47.029273 5103 ts_tablet_manager.cc:1367] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.182s user 0.008s sys 0.004s
I20240503 02:12:47.027686 6946 tablet.cc:1722] T efa7c0c5fc614c16985008b8fba54bab P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:47.030264 5103 raft_consensus.cc:348] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.031455 5103 raft_consensus.cc:374] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.031522 5103 raft_consensus.cc:729] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.031653 6901 raft_consensus.cc:482] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.031853 6901 raft_consensus.cc:504] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.032073 5103 consensus_queue.cc:260] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.032699 6901 leader_election.cc:288] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.028834 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 870) took 4811 ms. Trace:
I20240503 02:12:47.033068 5115 rpcz_store.cc:276] 0503 02:12:42.217299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.218651 (+ 1352us) service_pool.cc:225] Handling call
0503 02:12:46.921260 (+4702609us) ts_tablet_manager.cc:1120] Deleting tablet 42e80cc31f5c4908b82f2b030255f6af
0503 02:12:46.921268 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.964797 (+ 43529us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.004872 (+ 40075us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.028807 (+ 23935us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38750,"tablet-delete.queue_time_us":4702429}
I20240503 02:12:47.033769 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 869) took 4816 ms. Trace:
I20240503 02:12:47.033928 5115 rpcz_store.cc:276] 0503 02:12:42.217218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.218836 (+ 1618us) service_pool.cc:225] Handling call
0503 02:12:47.033586 (+4814750us) ts_tablet_manager.cc:1120] Deleting tablet d99a7e930a3947b4aae62424f11d7227
0503 02:12:47.033601 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.033747 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4814601}
I20240503 02:12:47.034392 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 872) took 4774 ms. Trace:
I20240503 02:12:47.034541 5115 rpcz_store.cc:276] 0503 02:12:42.259831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.269553 (+ 9722us) service_pool.cc:225] Handling call
0503 02:12:47.034266 (+4764713us) ts_tablet_manager.cc:1120] Deleting tablet 810d37e454cb4633a099c546925eb48c
0503 02:12:47.034276 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.034374 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4764309}
I20240503 02:12:46.953930 5159 tablet_replica.cc:330] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:46.970396 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb174825f1441f861ae6816137f855 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a271c33588f432a801b8cbc0fb79885) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.035410 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 871) took 4775 ms. Trace:
I20240503 02:12:47.035492 5115 rpcz_store.cc:276] 0503 02:12:42.259716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.273071 (+ 13355us) service_pool.cc:225] Handling call
0503 02:12:47.035292 (+4762221us) ts_tablet_manager.cc:1120] Deleting tablet b7d0640cea2a482e9974376ba479163b
0503 02:12:47.035301 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.035392 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4761792}
I20240503 02:12:47.035748 5115 tablet_replica.cc:330] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.035794 5159 raft_consensus.cc:2229] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.036015 5115 raft_consensus.cc:2229] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.036091 5159 raft_consensus.cc:2258] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.036239 5115 raft_consensus.cc:2258] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.036864 5115 ts_tablet_manager.cc:1875] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.037444 5159 ts_tablet_manager.cc:1875] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.038247 5105 tablet_bootstrap.cc:492] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.038390 5105 ts_tablet_manager.cc:1367] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.135s user 0.005s sys 0.003s
I20240503 02:12:47.039146 5105 raft_consensus.cc:348] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.039353 5105 raft_consensus.cc:374] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.039419 5105 raft_consensus.cc:729] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.039932 5103 ts_tablet_manager.cc:1398] T ec641e403ed843fda31dabd3920aeb68 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:47.039678 5105 consensus_queue.cc:260] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.040256 5103 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:47.041826 5103 tablet_bootstrap.cc:654] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.041798 5105 ts_tablet_manager.cc:1398] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:47.052486 4518 raft_consensus.cc:3037] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.013901 5114 ts_tablet_manager.cc:1888] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240503 02:12:47.054517 5114 log.cc:1196] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/f40706400f114def9e14e6d1d7114962
I20240503 02:12:47.054935 5114 ts_tablet_manager.cc:1909] T f40706400f114def9e14e6d1d7114962 P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
I20240503 02:12:47.055066 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b32c377006a24372859ed44d59cf90ee (table default.Table_80dedb822df54ccc9e63c27319fec3b9 [id=921e31bf406040b9a44409fdfbb79d2e]). Expected version 2 got 0
I20240503 02:12:47.055830 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 962) took 2520 ms. Trace:
I20240503 02:12:47.055963 5114 rpcz_store.cc:276] 0503 02:12:44.535285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.535530 (+ 245us) service_pool.cc:225] Handling call
0503 02:12:46.867896 (+2332366us) ts_tablet_manager.cc:1120] Deleting tablet f40706400f114def9e14e6d1d7114962
0503 02:12:46.867907 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:46.887550 (+ 19643us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.013783 (+126233us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.055788 (+ 42005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332133}
I20240503 02:12:47.056501 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]) successfully deleted
I20240503 02:12:47.057797 5106 tablet_bootstrap.cc:654] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.055848 4518 raft_consensus.cc:1261] T bf67d8c1d3ec4f758cca9745c42c7ee6 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Refusing update from remote peer 1f50719d94564844b15e1a9211d48a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:47.060024 5105 tablet_bootstrap.cc:492] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.060432 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 963) took 2525 ms. Trace:
I20240503 02:12:47.060539 5114 rpcz_store.cc:276] 0503 02:12:44.535374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.535801 (+ 427us) service_pool.cc:225] Handling call
0503 02:12:47.060289 (+2524488us) ts_tablet_manager.cc:1120] Deleting tablet 7f88d274b7654ba4942e757ba3302101
0503 02:12:47.060299 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.060409 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524353}
W20240503 02:12:47.061051 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 7f88d274b7654ba4942e757ba3302101 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f88d274b7654ba4942e757ba3302101 already in progress: creating tablet
I20240503 02:12:47.062377 5105 tablet_bootstrap.cc:654] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.065076 4268 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_637aa20655eb485f921eeebcadeee3eb [id=475bf67c3074494eb43e612a6897f57b] alter complete (version 2)
I20240503 02:12:47.066139 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 964) took 2530 ms. Trace:
I20240503 02:12:47.066275 5114 rpcz_store.cc:276] 0503 02:12:44.535459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.535955 (+ 496us) service_pool.cc:225] Handling call
0503 02:12:47.065937 (+2529982us) ts_tablet_manager.cc:1120] Deleting tablet bd4940fd3c16424eaaaef59ab50a04a8
0503 02:12:47.065955 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.066104 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529843}
W20240503 02:12:47.066780 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet bd4940fd3c16424eaaaef59ab50a04a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4940fd3c16424eaaaef59ab50a04a8 already in progress: creating tablet
I20240503 02:12:47.067101 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 961) took 2532 ms. Trace:
I20240503 02:12:47.067215 5114 rpcz_store.cc:276] 0503 02:12:44.535128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.564536 (+ 29408us) service_pool.cc:225] Handling call
0503 02:12:47.066819 (+2502283us) ts_tablet_manager.cc:1120] Deleting tablet 9003cc376aa149b59947bc32bd071163
0503 02:12:47.066929 (+ 110us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.067074 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502030}
W20240503 02:12:47.067663 4259 catalog_manager.cc:4810] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): delete failed for tablet 9003cc376aa149b59947bc32bd071163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9003cc376aa149b59947bc32bd071163 already in progress: creating tablet
I20240503 02:12:47.067749 5114 tablet_replica.cc:330] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:47.068046 5114 raft_consensus.cc:2229] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.068305 5114 raft_consensus.cc:2258] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.069144 5114 ts_tablet_manager.cc:1875] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:47.069386 6876 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 1338856135738044918) tablet 00000000000000000000000000000000 cur holder 0x61700007ee80
I20240503 02:12:47.072667 4270 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:47.073755 5103 tablet_bootstrap.cc:492] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:47.073904 5103 ts_tablet_manager.cc:1367] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20240503 02:12:47.074573 5103 raft_consensus.cc:348] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.074764 5103 raft_consensus.cc:374] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.074811 5103 raft_consensus.cc:729] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.074983 5103 consensus_queue.cc:260] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.075485 5103 ts_tablet_manager.cc:1398] T 94559cfc32704eae96b121f165686edc P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:47.075975 5103 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:47.076673 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet efa7c0c5fc614c16985008b8fba54bab (table default.Table_637aa20655eb485f921eeebcadeee3eb [id=475bf67c3074494eb43e612a6897f57b]). Expected version 2 got 0
I20240503 02:12:47.077519 5103 tablet_bootstrap.cc:654] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.047578 4527 raft_consensus.cc:2454] T c099a306a56643069f63e34a5f0d3dd7 P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:47.081732 5103 tablet_bootstrap.cc:492] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:47.081912 5103 ts_tablet_manager.cc:1367] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:47.082626 5103 raft_consensus.cc:348] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.082831 5103 raft_consensus.cc:374] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.082898 5103 raft_consensus.cc:729] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9f079230a468f90e5e3a58f254b91, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.083146 5103 consensus_queue.cc:260] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.083840 5103 ts_tablet_manager.cc:1398] T 0cc2ebfb11aa4840adcc35da73be75a2 P f4f9f079230a468f90e5e3a58f254b91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:47.084205 5103 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P f4f9f079230a468f90e5e3a58f254b91: Bootstrap starting.
I20240503 02:12:47.085480 6892 consensus_queue.cc:1035] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.087517 5103 tablet_bootstrap.cc:654] T a73ec6543b9a468492f3b97772ce43f9 P f4f9f079230a468f90e5e3a58f254b91: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.088995 6941 raft_consensus.cc:482] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.089120 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.089157 6941 raft_consensus.cc:504] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.089856 6941 leader_election.cc:288] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.090405 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aad7d6dcf2e4009872ce156c536f38e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:47.091004 4746 leader_election.cc:341] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.093537 4268 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet efa7c0c5fc614c16985008b8fba54bab (table default.Table_637aa20655eb485f921eeebcadeee3eb [id=475bf67c3074494eb43e612a6897f57b]). Expected version 2 got 0
I20240503 02:12:47.093770 4268 catalog_manager.cc:5576] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:47.094853 5114 ts_tablet_manager.cc:1888] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:47.095161 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce9d746acfe4e9aaea600293470d373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.095394 5114 log.cc:1196] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/79aae500e9a842ed90357f4a65fd37ba
I20240503 02:12:47.095896 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet ec641e403ed843fda31dabd3920aeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet d35138f04f224475af324e77aa1e0191) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:47.096328 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 367de4ff0a6f4abfa2924a070f387cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:47.100131 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aad7d6dcf2e4009872ce156c536f38e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:47.100392 5114 ts_tablet_manager.cc:1909] T 79aae500e9a842ed90357f4a65fd37ba P f4f9f079230a468f90e5e3a58f254b91: Deleting consensus metadata
W20240503 02:12:47.101027 4746 leader_election.cc:341] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.101187 4746 leader_election.cc:302] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:47.101464 6941 raft_consensus.cc:2726] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:47.101794 4462 consensus_peers.cc:485] T c1f39e8d17b44895bd5c979d82145ad9 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1f39e8d17b44895bd5c979d82145ad9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:47.102413 5114 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51106 (request call id 968) took 2081 ms. Trace:
I20240503 02:12:47.102939 5114 rpcz_store.cc:276] 0503 02:12:45.021494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.021569 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:47.067689 (+2046120us) ts_tablet_manager.cc:1120] Deleting tablet 79aae500e9a842ed90357f4a65fd37ba
0503 02:12:47.067700 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.091256 (+ 23556us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.094738 (+ 3482us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.102392 (+ 7654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2011577}
I20240503 02:12:47.105326 5115 ts_tablet_manager.cc:1888] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:47.105463 5115 log.cc:1196] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/64cdb861bc9f448698e243718136c303
I20240503 02:12:47.105861 5115 ts_tablet_manager.cc:1909] T 64cdb861bc9f448698e243718136c303 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.106662 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 873) took 4714 ms. Trace:
I20240503 02:12:47.106755 5115 rpcz_store.cc:276] 0503 02:12:42.392246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.392311 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:47.035702 (+4643391us) ts_tablet_manager.cc:1120] Deleting tablet 64cdb861bc9f448698e243718136c303
0503 02:12:47.035710 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.101145 (+ 65435us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.105200 (+ 4055us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.106639 (+ 1439us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4643199}
I20240503 02:12:47.107079 5115 tablet_replica.cc:330] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.107308 5115 raft_consensus.cc:2229] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.107617 5115 raft_consensus.cc:2258] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:47.108206 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 452d494e14c440e6805055ba333b4e7f (table default.Table_1f82b7031a824857bfc325fc0c6b1292 [id=3f70407631dd4d3c9a303411aaeaeaba], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 452d494e14c440e6805055ba333b4e7f consensus state has no leader
I20240503 02:12:47.108213 5115 ts_tablet_manager.cc:1875] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:47.109794 4608 leader_election.cc:341] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 043ba14eaa5b46319b0b88457d2a5f45
I20240503 02:12:47.110150 4608 leader_election.cc:302] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.110595 4608 leader_election.cc:302] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:47.111073 5105 tablet_bootstrap.cc:492] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.108841 4500 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:47.111234 6897 raft_consensus.cc:2781] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.109660 4491 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc2ebfb11aa4840adcc35da73be75a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:51106
I20240503 02:12:47.111356 6897 raft_consensus.cc:686] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.111402 5114 tablet_replica.cc:330] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: stopping tablet replica
I20240503 02:12:47.111601 5114 raft_consensus.cc:2229] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.111601 4259 catalog_manager.cc:4830] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): tablet 79aae500e9a842ed90357f4a65fd37ba (table default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]) successfully deleted
I20240503 02:12:47.111598 6897 consensus_queue.cc:237] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.111899 5105 ts_tablet_manager.cc:1367] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20240503 02:12:47.111927 5114 raft_consensus.cc:2258] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.112722 5105 raft_consensus.cc:348] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.112962 5105 raft_consensus.cc:374] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.113025 5105 raft_consensus.cc:729] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.113009 6899 raft_consensus.cc:2781] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.113150 6899 raft_consensus.cc:686] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.113257 5105 consensus_queue.cc:260] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.113422 6899 consensus_queue.cc:237] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.113768 5105 ts_tablet_manager.cc:1398] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:47.114049 5105 tablet_bootstrap.cc:492] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.115936 5105 tablet_bootstrap.cc:654] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.130038 5159 ts_tablet_manager.cc:1888] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.130316 5159 log.cc:1196] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/367de4ff0a6f4abfa2924a070f387cbd
I20240503 02:12:47.130786 5159 ts_tablet_manager.cc:1909] T 367de4ff0a6f4abfa2924a070f387cbd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.131714 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 951) took 3585 ms. Trace:
I20240503 02:12:47.132329 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet aa699a1dc3c34be59925560d5e11f49e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.132395 5159 rpcz_store.cc:276] 0503 02:12:43.546532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.546591 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:46.953872 (+3407281us) ts_tablet_manager.cc:1120] Deleting tablet 367de4ff0a6f4abfa2924a070f387cbd
0503 02:12:46.953885 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.048121 (+ 94236us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.129881 (+ 81760us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.131686 (+ 1805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3407077}
I20240503 02:12:47.132637 6948 raft_consensus.cc:482] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.132822 6948 raft_consensus.cc:504] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.133085 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 367de4ff0a6f4abfa2924a070f387cbd (table default.Table_cc2e1a713cc449a68a25796a8b26205a [id=2426f7ecae8b480789c148cbdffc5291]) successfully deleted
I20240503 02:12:47.133710 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 953) took 3504 ms. Trace:
I20240503 02:12:47.133776 6948 leader_election.cc:288] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.133870 5159 rpcz_store.cc:276] 0503 02:12:43.629197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.749019 (+119822us) service_pool.cc:225] Handling call
0503 02:12:47.133556 (+3384537us) ts_tablet_manager.cc:1120] Deleting tablet 0794df2bfc5e45db95cf4c745892e695
0503 02:12:47.133568 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.133689 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3384303}
W20240503 02:12:47.134498 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 0794df2bfc5e45db95cf4c745892e695 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0794df2bfc5e45db95cf4c745892e695 already in progress: creating tablet
I20240503 02:12:47.134898 5159 tablet_replica.cc:330] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.135147 5159 raft_consensus.cc:2229] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.135354 5159 raft_consensus.cc:2258] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.135978 5159 ts_tablet_manager.cc:1875] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.137665 5114 ts_tablet_manager.cc:1875] T 53b149e66a0446eeade190dd7f5d2bec P f4f9f079230a468f90e5e3a58f254b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.138351 4608 leader_election.cc:302] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.138710 6830 raft_consensus.cc:2781] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.138792 6830 raft_consensus.cc:482] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.138895 6830 raft_consensus.cc:3037] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:47.140002 4608 consensus_peers.cc:485] T 84b6e303e34a465caffed6c9e1e6bf74 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84b6e303e34a465caffed6c9e1e6bf74. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:47.140307 6901 consensus_queue.cc:1035] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.140733 6901 consensus_queue.cc:1035] T 1f3dbcbc572f4215b7f59f3750a4b9fa P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.141906 6901 consensus_queue.cc:1035] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.142702 6830 raft_consensus.cc:504] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.143311 6830 leader_election.cc:288] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.143960 4639 tablet_service.cc:1467] Processing CreateTablet for tablet 087fc9564e2f405b82069448757a2d8e (DEFAULT_TABLE table=default.Table_370e31b9745e49a69e323d187f951fc9 [id=38af125dc8b14378a4987f82a424b5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.144402 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087fc9564e2f405b82069448757a2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.145104 5159 ts_tablet_manager.cc:1888] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.145236 5159 log.cc:1196] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ec641e403ed843fda31dabd3920aeb68
I20240503 02:12:47.145604 5159 ts_tablet_manager.cc:1909] T ec641e403ed843fda31dabd3920aeb68 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.145956 4271 catalog_manager.cc:5484] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:47.147049 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 966) took 3325 ms. Trace:
I20240503 02:12:47.147152 5159 rpcz_store.cc:276] 0503 02:12:43.821182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.821995 (+ 813us) service_pool.cc:225] Handling call
0503 02:12:47.134844 (+3312849us) ts_tablet_manager.cc:1120] Deleting tablet ec641e403ed843fda31dabd3920aeb68
0503 02:12:47.134855 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.141031 (+ 6176us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.144983 (+ 3952us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.147023 (+ 2040us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3312474}
I20240503 02:12:47.147615 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ec641e403ed843fda31dabd3920aeb68 (table default.Table_e6e97943c4dd41638a2b090765b119a3 [id=e5a5a9bb92f34937b576e14ebe63c59c]) successfully deleted
I20240503 02:12:47.147946 5159 tablet_replica.cc:330] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.148196 5159 raft_consensus.cc:2229] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.148434 5159 raft_consensus.cc:2258] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.148599 5105 tablet_bootstrap.cc:492] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.148728 5105 ts_tablet_manager.cc:1367] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240503 02:12:47.149215 5159 ts_tablet_manager.cc:1875] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.149621 5105 raft_consensus.cc:348] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.149879 5105 raft_consensus.cc:374] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.149940 5105 raft_consensus.cc:729] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.150194 5105 consensus_queue.cc:260] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.139737 5103 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P f4f9f079230a468f90e5e3a58f254b91: No bootstrap required, opened a new log
I20240503 02:12:47.146600 4569 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:47.148087 4462 consensus_peers.cc:485] T 81d6d3ca80c54c629a5de82b06a7aac4 P f4f9f079230a468f90e5e3a58f254b91 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81d6d3ca80c54c629a5de82b06a7aac4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:47.158883 5105 ts_tablet_manager.cc:1398] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:12:47.159148 5106 tablet_bootstrap.cc:492] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.159262 5105 tablet_bootstrap.cc:492] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.159859 5103 ts_tablet_manager.cc:1367] T a73ec6543b9a468492f3b97772ce43f9 P f4f9f079230a468f90e5e3a58f254b91: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.005s
I20240503 02:12:47.160120 5106 ts_tablet_manager.cc:1367] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.001s
I20240503 02:12:47.160948 5105 tablet_bootstrap.cc:654] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.166200 5159 ts_tablet_manager.cc:1888] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.166460 5159 log.cc:1196] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/0cc2ebfb11aa4840adcc35da73be75a2
I20240503 02:12:47.166879 5159 ts_tablet_manager.cc:1909] T 0cc2ebfb11aa4840adcc35da73be75a2 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.167768 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 967) took 3345 ms. Trace:
I20240503 02:12:47.167938 5159 rpcz_store.cc:276] 0503 02:12:43.822662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.824388 (+ 1726us) service_pool.cc:225] Handling call
0503 02:12:47.147895 (+3323507us) ts_tablet_manager.cc:1120] Deleting tablet 0cc2ebfb11aa4840adcc35da73be75a2
0503 02:12:47.147903 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.152794 (+ 4891us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.166060 (+ 13266us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.167745 (+ 1685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3323276}
I20240503 02:12:47.168650 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 0cc2ebfb11aa4840adcc35da73be75a2 (table default.Table_0cc3309db13041969f6cbdf461a53f8f [id=60f853a8a91343cabaa48d3d929b1961]) successfully deleted
I20240503 02:12:47.169210 5159 tablet_replica.cc:330] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.169485 5159 raft_consensus.cc:2229] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.169452 4639 tablet_service.cc:1467] Processing CreateTablet for tablet c1eb672d91914513a4b04a9a5c70726a (DEFAULT_TABLE table=default.Table_62f0644508ee4e159ae0fd95cb3558c2 [id=f38b3ca16f5b4deb99967782dd73dfdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.169754 5159 raft_consensus.cc:2258] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.170158 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1eb672d91914513a4b04a9a5c70726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.167858 5106 raft_consensus.cc:348] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.177742 5106 raft_consensus.cc:374] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.178406 4648 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6dd1cc17942139bda7d85c875e16b (DEFAULT_TABLE table=default.Table_dbb9132d9f3441548f0e10b9818769f8 [id=54353980644249588665c95a13c842d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.179167 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6dd1cc17942139bda7d85c875e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.179785 5159 ts_tablet_manager.cc:1875] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.190119 5105 tablet_bootstrap.cc:492] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.190302 5105 ts_tablet_manager.cc:1367] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.004s
I20240503 02:12:47.191076 5105 raft_consensus.cc:348] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.191344 5105 raft_consensus.cc:374] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.191418 5105 raft_consensus.cc:729] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.191671 5105 consensus_queue.cc:260] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.192445 5106 raft_consensus.cc:729] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.192530 5159 ts_tablet_manager.cc:1888] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.192685 5159 log.cc:1196] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/ee1fac803dd84eee8722f0a14559faf6
I20240503 02:12:47.193032 5159 ts_tablet_manager.cc:1909] T ee1fac803dd84eee8722f0a14559faf6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.194124 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 972) took 3296 ms. Trace:
I20240503 02:12:47.194262 5159 rpcz_store.cc:276] 0503 02:12:43.897334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.897382 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:47.169148 (+3271766us) ts_tablet_manager.cc:1120] Deleting tablet ee1fac803dd84eee8722f0a14559faf6
0503 02:12:47.169161 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.188254 (+ 19093us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.192427 (+ 4173us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.194089 (+ 1662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3271558}
I20240503 02:12:47.194872 6941 raft_consensus.cc:482] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.192966 5106 consensus_queue.cc:260] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.193553 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3229f7010a4d458e8d7ee82f7df499 (DEFAULT_TABLE table=default.Table_2bc261e8aa5a478786fbc5296356f0d5 [id=c5dae91c96ca47f4b7aff913fbcb5494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.195005 6941 raft_consensus.cc:504] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.195415 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3229f7010a4d458e8d7ee82f7df499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.195677 6941 leader_election.cc:288] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.196743 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91" is_pre_election: true
I20240503 02:12:47.197146 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.197384 4667 raft_consensus.cc:2454] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:47.197693 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet ee1fac803dd84eee8722f0a14559faf6 (table default.Table_612f648735a34eff8dafc63f119da941 [id=f7a7a1dbe706435a934d25f99b3114cc]) successfully deleted
W20240503 02:12:47.198318 4746 leader_election.cc:341] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.198565 5159 tablet_replica.cc:330] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.198604 4746 leader_election.cc:302] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:47.198859 6814 raft_consensus.cc:482] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.198894 5159 raft_consensus.cc:2229] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.199015 6941 raft_consensus.cc:2781] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.199084 6941 raft_consensus.cc:482] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.199151 6941 raft_consensus.cc:3037] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.199977 5105 ts_tablet_manager.cc:1398] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 02:12:47.201983 6941 raft_consensus.cc:504] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.203078 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9f079230a468f90e5e3a58f254b91"
I20240503 02:12:47.194340 4648 tablet_service.cc:1467] Processing CreateTablet for tablet bea254eeb9b84fc283032704fb450d7f (DEFAULT_TABLE table=default.Table_bea552b4d4954b9b80d91b223a4349ce [id=1cb7f252528a439db33f7877dc8f2771]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:47.203696 4746 leader_election.cc:341] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.203812 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea254eeb9b84fc283032704fb450d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.204098 5105 tablet_bootstrap.cc:492] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.199023 6814 raft_consensus.cc:504] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.205806 6814 leader_election.cc:288] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.206933 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa699a1dc3c34be59925560d5e11f49e" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.207083 6941 leader_election.cc:288] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.207334 4638 tablet_service.cc:1467] Processing CreateTablet for tablet b7a366b6b96346e9b9dc454613db662e (DEFAULT_TABLE table=default.Table_6dd79385e5714ac1828bcf7463efbd5e [id=d8980a9ef229444d8f888769979b0f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.207702 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b743e99847bb47d1ae437c96789a8ea7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:47.207854 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a366b6b96346e9b9dc454613db662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.199141 5159 raft_consensus.cc:2258] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:47.208001 4746 leader_election.cc:341] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.207875 4667 raft_consensus.cc:3037] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.208580 5159 ts_tablet_manager.cc:1875] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.210893 4667 raft_consensus.cc:2454] T b743e99847bb47d1ae437c96789a8ea7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:47.211426 4746 leader_election.cc:302] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters: f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:47.211885 6941 raft_consensus.cc:2781] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.212523 6941 raft_consensus.cc:686] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:47.212872 6941 consensus_queue.cc:237] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.215193 5159 ts_tablet_manager.cc:1888] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.216570 5105 tablet_bootstrap.cc:654] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.217768 5159 log.cc:1196] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c757a3a979c2489daa621cf8ea293ea8
I20240503 02:12:47.218308 5159 ts_tablet_manager.cc:1909] T c757a3a979c2489daa621cf8ea293ea8 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.219154 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 973) took 3299 ms. Trace:
I20240503 02:12:47.219321 5159 rpcz_store.cc:276] 0503 02:12:43.919220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.922926 (+ 3706us) service_pool.cc:225] Handling call
0503 02:12:47.198506 (+3275580us) ts_tablet_manager.cc:1120] Deleting tablet c757a3a979c2489daa621cf8ea293ea8
0503 02:12:47.198519 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.211646 (+ 13127us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.215074 (+ 3428us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.219129 (+ 4055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3275289}
I20240503 02:12:47.220021 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c757a3a979c2489daa621cf8ea293ea8 (table default.Table_0920c4cdbad14eaa91462804e2bdff73 [id=ed2359e77c7c44208a625e7c4a672e3e]) successfully deleted
I20240503 02:12:47.220597 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 975) took 3193 ms. Trace:
I20240503 02:12:47.220726 5159 rpcz_store.cc:276] 0503 02:12:44.026785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.026842 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:47.220448 (+3193606us) ts_tablet_manager.cc:1120] Deleting tablet 151f925dcb304594b60c85d850b83533
0503 02:12:47.220461 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.220578 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3167991}
I20240503 02:12:47.221382 4638 tablet_service.cc:1467] Processing CreateTablet for tablet d96c2ce647614957804fc5689a0ee0a4 (DEFAULT_TABLE table=default.Table_e12eac6a8aaa48a9b27f9995e7e36b10 [id=cc88cd0e88ed4908869660446ffa7df3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 02:12:47.221515 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 151f925dcb304594b60c85d850b83533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 151f925dcb304594b60c85d850b83533 already in progress: creating tablet
I20240503 02:12:47.222062 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 977) took 3185 ms. Trace:
I20240503 02:12:47.222273 5159 rpcz_store.cc:276] 0503 02:12:44.037111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.037168 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:47.221884 (+3184716us) ts_tablet_manager.cc:1120] Deleting tablet 3ee32345fb9349f19af09a3447c4caad
0503 02:12:47.221899 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.222041 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3163790}
W20240503 02:12:47.222937 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 3ee32345fb9349f19af09a3447c4caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee32345fb9349f19af09a3447c4caad already in progress: creating tablet
I20240503 02:12:47.223506 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 978) took 3181 ms. Trace:
I20240503 02:12:47.223662 5159 rpcz_store.cc:276] 0503 02:12:44.041957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.042019 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:47.223340 (+3181321us) ts_tablet_manager.cc:1120] Deleting tablet 2848c837598c49628c11374d3c875b3e
0503 02:12:47.223355 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.223486 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3165168}
W20240503 02:12:47.224256 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 2848c837598c49628c11374d3c875b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2848c837598c49628c11374d3c875b3e already in progress: creating tablet
I20240503 02:12:47.224778 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 976) took 3188 ms. Trace:
I20240503 02:12:47.224941 5159 rpcz_store.cc:276] 0503 02:12:44.035915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.036050 (+ 135us) service_pool.cc:225] Handling call
0503 02:12:47.224622 (+3188572us) ts_tablet_manager.cc:1120] Deleting tablet e82190dfb1b1418b9f20bf4d0c2c2409
0503 02:12:47.224637 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.224758 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3166321}
W20240503 02:12:47.225481 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet e82190dfb1b1418b9f20bf4d0c2c2409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82190dfb1b1418b9f20bf4d0c2c2409 already in progress: creating tablet
I20240503 02:12:47.225549 5106 ts_tablet_manager.cc:1398] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20240503 02:12:47.225852 5159 tablet_replica.cc:330] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.226152 5159 raft_consensus.cc:2229] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.226917 5159 raft_consensus.cc:2258] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.226992 6814 raft_consensus.cc:482] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.227121 6814 raft_consensus.cc:504] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.227703 5105 tablet_bootstrap.cc:492] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.227841 6814 leader_election.cc:288] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.229171 5159 ts_tablet_manager.cc:1875] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.232426 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 68a60272d2e14853a6f5e6c7d07f8b74 (DEFAULT_TABLE table=default.Table_6d1f952d30ef4ea4ba59b6b7ebf7548d [id=e1031d4cc15c4aacbe97975af2991ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.227831 5105 ts_tablet_manager.cc:1367] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240503 02:12:47.234198 5105 raft_consensus.cc:348] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.234411 5105 raft_consensus.cc:374] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.234470 5105 raft_consensus.cc:729] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.235985 5105 consensus_queue.cc:260] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.236169 5159 ts_tablet_manager.cc:1888] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.236256 5159 log.cc:1196] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/dbfe6b2b9cab4517ae7cabec4796c9ce
I20240503 02:12:47.236897 5159 ts_tablet_manager.cc:1909] T dbfe6b2b9cab4517ae7cabec4796c9ce P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.237223 5105 ts_tablet_manager.cc:1398] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:47.237917 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 979) took 3152 ms. Trace:
I20240503 02:12:47.238116 5159 rpcz_store.cc:276] 0503 02:12:44.085980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.086050 (+ 70us) service_pool.cc:225] Handling call
0503 02:12:47.225800 (+3139750us) ts_tablet_manager.cc:1120] Deleting tablet dbfe6b2b9cab4517ae7cabec4796c9ce
0503 02:12:47.225810 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.232502 (+ 6692us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.236061 (+ 3559us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.237883 (+ 1822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3139273}
I20240503 02:12:47.238524 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d795ee01c417dbdeda41fd6853dd7" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.238814 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet dbfe6b2b9cab4517ae7cabec4796c9ce (table default.Table_8ac2cdb3ebec4a2795b7fa6e47dc56b8 [id=18e4f81f1509432ab8241d5d28e7a1e4]) successfully deleted
I20240503 02:12:47.239307 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 980) took 3081 ms. Trace:
I20240503 02:12:47.239526 5106 tablet_bootstrap.cc:492] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.239452 5159 rpcz_store.cc:276] 0503 02:12:44.158343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.158394 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:47.239158 (+3080764us) ts_tablet_manager.cc:1120] Deleting tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2
0503 02:12:47.239169 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.239290 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3080526}
W20240503 02:12:47.240131 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 already in progress: creating tablet
I20240503 02:12:47.240564 5159 tablet_replica.cc:330] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.240818 5159 raft_consensus.cc:2229] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.241098 5159 pending_rounds.cc:62] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Trying to abort 1 pending ops.
I20240503 02:12:47.241235 5106 tablet_bootstrap.cc:654] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:47.241199 4746 leader_election.cc:341] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.241209 5159 pending_rounds.cc:68] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420889532600320 op_type: NO_OP noop_request { }
I20240503 02:12:47.241375 5159 raft_consensus.cc:2866] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:47.242614 4647 tablet_service.cc:1467] Processing CreateTablet for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d (DEFAULT_TABLE table=default.Table_552f6a00a910447e9ee59068d1d3908f [id=bf6c7fd1504f4ed8a53148da7e7806f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.242648 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet dee34be89bda450296b916163bb10c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.243072 4644 tablet_service.cc:1467] Processing CreateTablet for tablet dbd23f93e788468b99dba3ffa1398450 (DEFAULT_TABLE table=default.Table_0f2571bf88ac4a23b6b806c2ae05671b [id=8eefd98351bc4f1ca157b595ddda25a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.243170 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 94559cfc32704eae96b121f165686edc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.244710 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ab8dbf1a844ca39e9dea1c3b98b00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.245536 5159 raft_consensus.cc:2258] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.246639 5159 ts_tablet_manager.cc:1875] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.247172 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf10c48d165047668fbd5ca17f917fe3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:47.247507 4812 raft_consensus.cc:2454] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:47.248163 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 42e80cc31f5c4908b82f2b030255f6af (table default.Table_be851867e140416c8921098eaa348201 [id=04d5d63e679c47ce9a72f1ffeeeaf506]) successfully deleted
W20240503 02:12:47.248667 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet d99a7e930a3947b4aae62424f11d7227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d99a7e930a3947b4aae62424f11d7227 already in progress: creating tablet
W20240503 02:12:47.248906 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 810d37e454cb4633a099c546925eb48c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810d37e454cb4633a099c546925eb48c already in progress: creating tablet
I20240503 02:12:47.249370 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd23f93e788468b99dba3ffa1398450. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:47.249433 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b7d0640cea2a482e9974376ba479163b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d0640cea2a482e9974376ba479163b already in progress: creating tablet
I20240503 02:12:47.255616 5159 ts_tablet_manager.cc:1888] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.255877 5159 log.cc:1196] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/73965590a0ea4bffae6e99fa1367fad0
I20240503 02:12:47.256415 5159 ts_tablet_manager.cc:1909] T 73965590a0ea4bffae6e99fa1367fad0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.259896 6899 raft_consensus.cc:482] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.260109 6899 raft_consensus.cc:504] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.260854 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a60272d2e14853a6f5e6c7d07f8b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.261063 6899 leader_election.cc:288] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.263085 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 981) took 3101 ms. Trace:
I20240503 02:12:47.263280 5159 rpcz_store.cc:276] 0503 02:12:44.161926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.161980 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:47.240510 (+3078530us) ts_tablet_manager.cc:1120] Deleting tablet 73965590a0ea4bffae6e99fa1367fad0
0503 02:12:47.240522 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.252057 (+ 11535us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.255490 (+ 3433us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.263054 (+ 7564us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"tablet-delete.queue_time_us":3078356}
I20240503 02:12:47.264477 6814 raft_consensus.cc:482] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.264515 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 73965590a0ea4bffae6e99fa1367fad0 (table default.Table_9709e189b5424f23a7663146d1a912ac [id=5dfeefb7aef7484d8c5c9df2189f8932]) successfully deleted
I20240503 02:12:47.264644 6814 raft_consensus.cc:504] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.265058 5159 tablet_replica.cc:330] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.265319 6814 leader_election.cc:288] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.265317 5159 raft_consensus.cc:2229] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.265686 5159 raft_consensus.cc:2258] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.266106 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce4b33f90c40c48cf0a538a1a7fde4" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.266527 5159 ts_tablet_manager.cc:1875] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:47.266674 4746 leader_election.cc:341] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.272522 5159 ts_tablet_manager.cc:1888] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.272727 5159 log.cc:1196] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b84856917cb84c2a853c9a8a80246704
I20240503 02:12:47.273129 5159 ts_tablet_manager.cc:1909] T b84856917cb84c2a853c9a8a80246704 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.274159 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 982) took 3076 ms. Trace:
I20240503 02:12:47.274330 5159 rpcz_store.cc:276] 0503 02:12:44.197843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.197914 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:47.265004 (+3067090us) ts_tablet_manager.cc:1120] Deleting tablet b84856917cb84c2a853c9a8a80246704
0503 02:12:47.265015 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.269604 (+ 4589us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.272426 (+ 2822us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.274135 (+ 1709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3066837}
I20240503 02:12:47.238346 5105 tablet_bootstrap.cc:492] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.275239 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b84856917cb84c2a853c9a8a80246704 (table default.Table_8a7d18f84dac48edb23e19370f9fdb8f [id=83ba736efe7045f194563c021b282a5c]) successfully deleted
I20240503 02:12:47.276468 5105 tablet_bootstrap.cc:654] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.282428 5106 tablet_bootstrap.cc:492] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.282701 5106 ts_tablet_manager.cc:1367] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240503 02:12:47.283478 5106 raft_consensus.cc:348] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.285351 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 64cdb861bc9f448698e243718136c303 (table default.Table_230a1a13b4584a188a4235462025db5e [id=6fb8f01b6f4d46f883d05d680f5c0ec7]) successfully deleted
I20240503 02:12:47.290465 5106 raft_consensus.cc:374] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.291532 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a1d5b2de8d4d3ba5190750545c9f7b" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:47.291746 4807 raft_consensus.cc:3037] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.292963 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 983) took 3093 ms. Trace:
I20240503 02:12:47.293213 5159 rpcz_store.cc:276] 0503 02:12:44.199329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.199392 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:47.285788 (+3086396us) ts_tablet_manager.cc:1120] Deleting tablet b9470480a9de45209044f8e8c461519a
0503 02:12:47.285806 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.292941 (+ 7135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3086177}
I20240503 02:12:47.294142 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 984) took 3094 ms. Trace:
I20240503 02:12:47.294600 5159 rpcz_store.cc:276] 0503 02:12:44.199914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.199965 (+ 51us) service_pool.cc:225] Handling call
0503 02:12:47.293879 (+3093914us) ts_tablet_manager.cc:1120] Deleting tablet 8d1ac81e1a6f41cf974bb30e6b74eea2
0503 02:12:47.293894 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.294120 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3093739}
W20240503 02:12:47.295253 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 already in progress: creating tablet
I20240503 02:12:47.295480 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 985) took 3091 ms. Trace:
I20240503 02:12:47.295640 5159 rpcz_store.cc:276] 0503 02:12:44.204412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.204478 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:47.295207 (+3090729us) ts_tablet_manager.cc:1120] Deleting tablet 9d900a0dc6c34076a0993210693a57a0
0503 02:12:47.295222 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.295357 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3090534}
W20240503 02:12:47.296274 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9d900a0dc6c34076a0993210693a57a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d900a0dc6c34076a0993210693a57a0 already in progress: creating tablet
I20240503 02:12:47.296346 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 986) took 3086 ms. Trace:
I20240503 02:12:47.301321 5159 rpcz_store.cc:276] 0503 02:12:44.209943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.210002 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:47.296186 (+3086184us) ts_tablet_manager.cc:1120] Deleting tablet 7c19263f06ec41d7a6fe6a8df7f88f8f
0503 02:12:47.296201 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.296326 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085991}
W20240503 02:12:47.303323 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c19263f06ec41d7a6fe6a8df7f88f8f already in progress: creating tablet
I20240503 02:12:47.300395 4807 raft_consensus.cc:2454] T 83a1d5b2de8d4d3ba5190750545c9f7b P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:47.303395 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 987) took 3092 ms. Trace:
I20240503 02:12:47.304348 5159 rpcz_store.cc:276] 0503 02:12:44.211198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.211252 (+ 54us) service_pool.cc:225] Handling call
0503 02:12:47.303156 (+3091904us) ts_tablet_manager.cc:1120] Deleting tablet 89f3232d509f4cb39eb8923fe1d120a7
0503 02:12:47.303175 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.303369 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3091701}
W20240503 02:12:47.305286 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 89f3232d509f4cb39eb8923fe1d120a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f3232d509f4cb39eb8923fe1d120a7 already in progress: creating tablet
I20240503 02:12:47.305809 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 989) took 2903 ms. Trace:
I20240503 02:12:47.296499 6899 raft_consensus.cc:482] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.306264 5159 rpcz_store.cc:276] 0503 02:12:44.402190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.402282 (+ 92us) service_pool.cc:225] Handling call
0503 02:12:47.305351 (+2903069us) ts_tablet_manager.cc:1120] Deleting tablet 1b13ea83ffa54dcbbfebaaa76e1686a9
0503 02:12:47.305367 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.305782 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902817}
I20240503 02:12:47.306470 6899 raft_consensus.cc:504] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.307296 6899 leader_election.cc:288] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:47.307788 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 already in progress: creating tablet
I20240503 02:12:47.307934 5106 raft_consensus.cc:729] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.308233 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 993) took 2826 ms. Trace:
I20240503 02:12:47.308297 5106 consensus_queue.cc:260] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.310513 5106 ts_tablet_manager.cc:1398] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:12:47.311503 5106 tablet_bootstrap.cc:492] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.313084 5159 rpcz_store.cc:276] 0503 02:12:44.482034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.482918 (+ 884us) service_pool.cc:225] Handling call
0503 02:12:47.307871 (+2824953us) ts_tablet_manager.cc:1120] Deleting tablet 7219ce14a9ae4488b97914b9d02995f9
0503 02:12:47.308060 (+ 189us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.308214 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824751}
W20240503 02:12:47.313948 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7219ce14a9ae4488b97914b9d02995f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7219ce14a9ae4488b97914b9d02995f9 already in progress: creating tablet
I20240503 02:12:47.314369 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1005) took 2791 ms. Trace:
I20240503 02:12:47.314998 6973 tablet.cc:1722] T efa7c0c5fc614c16985008b8fba54bab P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:47.314625 5159 rpcz_store.cc:276] 0503 02:12:44.522787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.522835 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:47.313935 (+2791100us) ts_tablet_manager.cc:1120] Deleting tablet 7f88d274b7654ba4942e757ba3302101
0503 02:12:47.313945 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.314347 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2790865}
W20240503 02:12:47.315732 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7f88d274b7654ba4942e757ba3302101 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f88d274b7654ba4942e757ba3302101 already in progress: creating tablet
I20240503 02:12:47.316246 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1006) took 2792 ms. Trace:
I20240503 02:12:47.316397 5159 rpcz_store.cc:276] 0503 02:12:44.523367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.523435 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:47.316098 (+2792663us) ts_tablet_manager.cc:1120] Deleting tablet bd4940fd3c16424eaaaef59ab50a04a8
0503 02:12:47.316121 (+ 23us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.316227 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2792458}
W20240503 02:12:47.317000 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet bd4940fd3c16424eaaaef59ab50a04a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4940fd3c16424eaaaef59ab50a04a8 already in progress: creating tablet
I20240503 02:12:47.317978 5106 tablet_bootstrap.cc:654] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.322508 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1003) took 2800 ms. Trace:
I20240503 02:12:47.322635 5159 rpcz_store.cc:276] 0503 02:12:44.522050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.524667 (+ 2617us) service_pool.cc:225] Handling call
0503 02:12:47.317362 (+2792695us) ts_tablet_manager.cc:1120] Deleting tablet 9003cc376aa149b59947bc32bd071163
0503 02:12:47.322342 (+ 4980us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.322483 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2792509}
I20240503 02:12:47.322947 5159 tablet_replica.cc:330] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.323196 5159 raft_consensus.cc:2229] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.324319 5159 raft_consensus.cc:2258] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:47.325568 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 9003cc376aa149b59947bc32bd071163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9003cc376aa149b59947bc32bd071163 already in progress: creating tablet
I20240503 02:12:47.326391 5159 ts_tablet_manager.cc:1875] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:47.327025 4608 leader_election.cc:341] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.327356 5106 tablet_bootstrap.cc:492] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.327493 6899 consensus_queue.cc:1035] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.328343 5106 ts_tablet_manager.cc:1367] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:12:47.329062 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94559cfc32704eae96b121f165686edc" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:47.329283 4809 raft_consensus.cc:2454] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:47.329234 5106 raft_consensus.cc:348] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.329408 5106 raft_consensus.cc:374] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.329478 5106 raft_consensus.cc:729] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.329677 5106 consensus_queue.cc:260] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.330257 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee34be89bda450296b916163bb10c46" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:47.330476 4809 raft_consensus.cc:2454] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:47.332079 5106 ts_tablet_manager.cc:1398] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:47.332669 6899 raft_consensus.cc:482] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.332803 6899 raft_consensus.cc:504] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:47.333104 4608 leader_election.cc:341] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 8d0494ee4bd746069bfaf39bfb46687d
I20240503 02:12:47.333446 6899 leader_election.cc:288] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.333804 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet c757a3a979c2489daa621cf8ea293ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:57782
W20240503 02:12:47.334430 4608 leader_election.cc:341] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 43119471f91c4a1ba900f267c88db9a6
W20240503 02:12:47.334623 4608 leader_election.cc:341] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 4b4972057fae45168756d7c28ddbe2bf
I20240503 02:12:47.334774 4608 leader_election.cc:302] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.336457 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67857b858c4719b498291c52b7aceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:23 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.349620 5115 ts_tablet_manager.cc:1888] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.350090 5115 log.cc:1196] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/76b01907833d44f6a000a4de3a2a6e96
I20240503 02:12:47.350596 5115 ts_tablet_manager.cc:1909] T 76b01907833d44f6a000a4de3a2a6e96 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.351565 6948 raft_consensus.cc:2781] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.351655 6948 raft_consensus.cc:482] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.351718 6948 raft_consensus.cc:3037] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.354641 6948 raft_consensus.cc:504] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.355042 5105 tablet_bootstrap.cc:492] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.355290 5105 ts_tablet_manager.cc:1367] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.015s
I20240503 02:12:47.355361 6948 leader_election.cc:288] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.356177 5105 raft_consensus.cc:348] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.356469 5105 raft_consensus.cc:374] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.356529 5105 raft_consensus.cc:729] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.356767 5105 consensus_queue.cc:260] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.357314 5105 ts_tablet_manager.cc:1398] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:47.357585 5105 tablet_bootstrap.cc:492] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:47.357749 4608 consensus_peers.cc:485] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73965590a0ea4bffae6e99fa1367fad0. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:47.358049 4608 consensus_peers.cc:485] T dbfe6b2b9cab4517ae7cabec4796c9ce P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbfe6b2b9cab4517ae7cabec4796c9ce. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:47.358449 4608 leader_election.cc:341] T b84856917cb84c2a853c9a8a80246704 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: b84856917cb84c2a853c9a8a80246704
I20240503 02:12:47.359900 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 6a11c306232c4e49ab02be0f195df935 (table default.Table_aa8c3d142c5746cc83cb777a33f38060 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]). Expected version 1 got 0
I20240503 02:12:47.360231 4608 leader_election.cc:302] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.360478 6948 raft_consensus.cc:2781] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.360687 6948 raft_consensus.cc:686] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.360955 6948 consensus_queue.cc:237] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.362744 5159 ts_tablet_manager.cc:1888] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240503 02:12:47.362869 5159 log.cc:1196] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/f40706400f114def9e14e6d1d7114962
I20240503 02:12:47.363231 5159 ts_tablet_manager.cc:1909] T f40706400f114def9e14e6d1d7114962 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.363520 6946 tablet.cc:1722] T 8c27e37ebd0f49cdba7653548acbe1ea P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:47.367748 6946 tablet.cc:1722] T bf67d8c1d3ec4f758cca9745c42c7ee6 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:47.372125 5106 tablet_bootstrap.cc:492] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.374233 5106 tablet_bootstrap.cc:654] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.374526 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 875) took 4982 ms. Trace:
I20240503 02:12:47.377792 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1004) took 2855 ms. Trace:
I20240503 02:12:47.377938 5159 rpcz_store.cc:276] 0503 02:12:44.522576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.524885 (+ 2309us) service_pool.cc:225] Handling call
0503 02:12:47.322899 (+2798014us) ts_tablet_manager.cc:1120] Deleting tablet f40706400f114def9e14e6d1d7114962
0503 02:12:47.322909 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.350716 (+ 27807us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.362623 (+ 11907us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.377757 (+ 15134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2797808}
I20240503 02:12:47.378504 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]) successfully deleted
I20240503 02:12:47.380585 5159 tablet_replica.cc:330] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.380824 5159 raft_consensus.cc:2229] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.381042 5159 raft_consensus.cc:2258] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.381639 5159 ts_tablet_manager.cc:1875] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.381511 5115 rpcz_store.cc:276] 0503 02:12:42.392457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.392845 (+ 388us) service_pool.cc:225] Handling call
0503 02:12:47.107024 (+4714179us) ts_tablet_manager.cc:1120] Deleting tablet 76b01907833d44f6a000a4de3a2a6e96
0503 02:12:47.107036 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.279349 (+172313us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.349479 (+ 70130us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.374500 (+ 25021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4713993}
I20240503 02:12:47.382619 5115 tablet_replica.cc:330] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.384020 5115 raft_consensus.cc:2229] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.384301 5115 raft_consensus.cc:2258] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.384830 5105 tablet_bootstrap.cc:654] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.382174 4608 leader_election.cc:302] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.385188 6899 raft_consensus.cc:2781] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.385282 6899 raft_consensus.cc:482] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.385353 6899 raft_consensus.cc:3037] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.386358 4608 leader_election.cc:302] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.386668 6841 raft_consensus.cc:2781] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.386790 6841 raft_consensus.cc:482] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.386932 6841 raft_consensus.cc:3037] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.388592 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 6a11c306232c4e49ab02be0f195df935 (table default.Table_aa8c3d142c5746cc83cb777a33f38060 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]). Expected version 1 got 0
I20240503 02:12:47.388969 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:47.389232 4267 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:47.390023 4267 catalog_manager.cc:5484] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:47.390760 5115 ts_tablet_manager.cc:1875] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.401110 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf10c48d165047668fbd5ca17f917fe3" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:47.401346 4809 raft_consensus.cc:3037] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.401561 5159 ts_tablet_manager.cc:1888] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.401675 5159 log.cc:1196] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/94559cfc32704eae96b121f165686edc
I20240503 02:12:47.401973 5159 ts_tablet_manager.cc:1909] T 94559cfc32704eae96b121f165686edc P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.402760 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1007) took 2872 ms. Trace:
I20240503 02:12:47.402860 5159 rpcz_store.cc:276] 0503 02:12:44.530460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.531100 (+ 640us) service_pool.cc:225] Handling call
0503 02:12:47.380524 (+2849424us) ts_tablet_manager.cc:1120] Deleting tablet 94559cfc32704eae96b121f165686edc
0503 02:12:47.380536 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.391177 (+ 10641us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.401440 (+ 10263us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.402731 (+ 1291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2849170}
I20240503 02:12:47.403419 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 94559cfc32704eae96b121f165686edc (table default.Table_0ea8c89cbb9e45d9af49c75e9921666e [id=6c3e164d6e3b4ca891ff750a862edbab]) successfully deleted
I20240503 02:12:47.404151 4809 raft_consensus.cc:2454] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:47.406172 6973 tablet.cc:1722] T 8c27e37ebd0f49cdba7653548acbe1ea P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:47.412477 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b8474c841cb64373a223ad76b43ecf5f with delete_type TABLET_DATA_DELETED (replaced by tablet 73198ec769a9486da3f85e8fcbef8e78) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.412431 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.412887 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.413650 5159 tablet_replica.cc:330] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.413872 5159 raft_consensus.cc:2229] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.414042 5159 raft_consensus.cc:2258] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.414099 6973 tablet.cc:1722] T bf67d8c1d3ec4f758cca9745c42c7ee6 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:47.414757 5159 ts_tablet_manager.cc:1875] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.414763 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 76b01907833d44f6a000a4de3a2a6e96 (table default.Table_247b02731e084bc5985d5f3d2dbac760 [id=dd626e59f9904c8b92f8b02de4fdeaa1]) successfully deleted
I20240503 02:12:47.417507 6905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:47.417738 6905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
I20240503 02:12:47.418821 6905 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
W20240503 02:12:47.420733 6227 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.67:45417 (request call id 94) took 1542 ms (client timeout 1000 ms). Trace:
W20240503 02:12:47.420850 6227 rpcz_store.cc:269] 0503 02:12:45.878025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.881220 (+ 3195us) service_pool.cc:225] Handling call
0503 02:12:47.419330 (+1538110us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x616000040654. stack: 00007f6afea47756 00007f6afea47510 00007f6afda7a48d 0000000000955804 00000000009557cd 00007f6b0d58715d 00007f6b0d587253 00007f6b0a2b8d41 00007f6b11ea534d 00007f6b1202c668 00007f6b09020786 00007f6b090204b5 00007f6b0574f04e 00007f6b0574df2e 00007f6b05754405 00007f6b05754d2f
0503 02:12:47.420662 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3535261568}
I20240503 02:12:47.421079 5159 ts_tablet_manager.cc:1888] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.421209 5159 log.cc:1196] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d607da9689c14e51add9178a48b35913
I20240503 02:12:47.421537 5159 ts_tablet_manager.cc:1909] T d607da9689c14e51add9178a48b35913 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.422403 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1008) took 2848 ms. Trace:
I20240503 02:12:47.422441 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1245 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:47.422495 5159 rpcz_store.cc:276] 0503 02:12:44.574240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.602360 (+ 28120us) service_pool.cc:225] Handling call
0503 02:12:47.413592 (+2811232us) ts_tablet_manager.cc:1120] Deleting tablet d607da9689c14e51add9178a48b35913
0503 02:12:47.413604 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.417891 (+ 4287us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.420907 (+ 3016us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.422378 (+ 1471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2810659}
I20240503 02:12:47.422997 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet d607da9689c14e51add9178a48b35913 (table default.Table_a9cfba004dfe4c7e8def7c54b105546f [id=9de9b4171d6e4a3583e141004418b113]) successfully deleted
W20240503 02:12:47.423321 6228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.66:56341 (request call id 52) took 1522 ms (client timeout 1000 ms). Trace:
W20240503 02:12:47.423441 6228 rpcz_store.cc:269] 0503 02:12:45.900550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.900707 (+ 157us) service_pool.cc:225] Handling call
0503 02:12:47.421673 (+1520966us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x616000040654. stack: 00007f6afea47756 00007f6afea47510 00007f6afda7a48d 0000000000955804 00000000009557cd 00007f6b0d58715d 00007f6b0d587253 00007f6b0a2b8d41 00007f6b11ea534d 00007f6b1202c668 00007f6b09020786 00007f6b090204b5 00007f6b0574f04e 00007f6b0574df2e 00007f6b05754405 00007f6b05754d2f
0503 02:12:47.423278 (+ 1605us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3495055616}
I20240503 02:12:47.425607 5159 tablet_replica.cc:330] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.425633 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1245 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:47.425823 5159 raft_consensus.cc:2229] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.425994 5159 raft_consensus.cc:2258] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.426633 5159 ts_tablet_manager.cc:1875] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.432983 5159 ts_tablet_manager.cc:1888] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.433108 5159 log.cc:1196] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3478033c4ed94268a2436a2942c27a68
I20240503 02:12:47.433427 5159 ts_tablet_manager.cc:1909] T 3478033c4ed94268a2436a2942c27a68 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.432884 4271 catalog_manager.cc:5484] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:47.434262 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1012) took 2437 ms. Trace:
I20240503 02:12:47.434372 5159 rpcz_store.cc:276] 0503 02:12:44.996799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.999995 (+ 3196us) service_pool.cc:225] Handling call
0503 02:12:47.425539 (+2425544us) ts_tablet_manager.cc:1120] Deleting tablet 3478033c4ed94268a2436a2942c27a68
0503 02:12:47.425554 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.429739 (+ 4185us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.432822 (+ 3083us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.434239 (+ 1417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2425276}
I20240503 02:12:47.434922 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3478033c4ed94268a2436a2942c27a68 (table default.Table_9166690fbabf48e0b6a2cafbbb312ccd [id=9e76a01c973c4144b40a4dae43de71f1]) successfully deleted
I20240503 02:12:47.434986 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
I20240503 02:12:47.435436 5159 tablet_replica.cc:330] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.435653 5159 raft_consensus.cc:2229] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.435874 5159 raft_consensus.cc:2258] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.436486 5159 ts_tablet_manager.cc:1875] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.437351 6180 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:47.438014 5105 tablet_bootstrap.cc:492] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.438189 5105 ts_tablet_manager.cc:1367] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20240503 02:12:47.438931 5105 raft_consensus.cc:348] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.439103 5105 raft_consensus.cc:374] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.439158 5105 raft_consensus.cc:729] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.439435 5105 consensus_queue.cc:260] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.439972 5105 ts_tablet_manager.cc:1398] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:47.441211 5105 tablet_bootstrap.cc:492] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.443003 5159 ts_tablet_manager.cc:1888] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:47.443096 5159 log.cc:1196] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/79aae500e9a842ed90357f4a65fd37ba
I20240503 02:12:47.443444 5159 ts_tablet_manager.cc:1909] T 79aae500e9a842ed90357f4a65fd37ba P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.393978 6899 raft_consensus.cc:504] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.445716 6899 leader_election.cc:288] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.446846 4783 tablet_service.cc:1467] Processing CreateTablet for tablet c70d86c941f2445fbd6ff8b0be7e4043 (DEFAULT_TABLE table=default.Table_4511fe14eef348aa8f91987405ac7bfb [id=562a3fed23794d9d8ca6ae700b979883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.447983 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1013) took 2426 ms. Trace:
I20240503 02:12:47.449249 5105 tablet_bootstrap.cc:654] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.450785 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70d86c941f2445fbd6ff8b0be7e4043. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:47.451087 6226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.17.65:46199 (request call id 88) took 1588 ms (client timeout 1000 ms). Trace:
I20240503 02:12:47.448074 5159 rpcz_store.cc:276] 0503 02:12:45.021068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.021741 (+ 673us) service_pool.cc:225] Handling call
0503 02:12:47.435377 (+2413636us) ts_tablet_manager.cc:1120] Deleting tablet 79aae500e9a842ed90357f4a65fd37ba
0503 02:12:47.435390 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.439845 (+ 4455us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.442891 (+ 3046us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.447960 (+ 5069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2276472}
I20240503 02:12:47.453567 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1036) took 2018 ms. Trace:
W20240503 02:12:47.453485 6226 rpcz_store.cc:269] 0503 02:12:45.862504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.862703 (+ 199us) service_pool.cc:225] Handling call
0503 02:12:47.434689 (+1571986us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x616000040654. stack: 00007f6afea47756 00007f6afea47510 00007f6afda7a48d 0000000000955804 00000000009557cd 00007f6b0d58715d 00007f6b0d587253 00007f6b0a2b8d41 00007f6b11ea534d 00007f6b1202c668 00007f6b09020786 00007f6b090204b5 00007f6b0574f04e 00007f6b0574df2e 00007f6b05754405 00007f6b05754d2f
0503 02:12:47.451036 (+ 16347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3613049088}
I20240503 02:12:47.453670 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 79aae500e9a842ed90357f4a65fd37ba (table default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]) successfully deleted
I20240503 02:12:47.453670 5159 rpcz_store.cc:276] 0503 02:12:45.434841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.434885 (+ 44us) service_pool.cc:225] Handling call
0503 02:12:47.453414 (+2018529us) ts_tablet_manager.cc:1120] Deleting tablet ffef15b7e51840fea91b5349a1724cc5
0503 02:12:47.453426 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.453546 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018279}
I20240503 02:12:47.454016 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1035) took 2019 ms. Trace:
I20240503 02:12:47.454092 5159 rpcz_store.cc:276] 0503 02:12:45.434757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.434806 (+ 49us) service_pool.cc:225] Handling call
0503 02:12:47.453905 (+2019099us) ts_tablet_manager.cc:1120] Deleting tablet db6fa251094842b9a629fdb67e20690b
0503 02:12:47.453914 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.454001 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018624}
I20240503 02:12:47.455998 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1037) took 1983 ms. Trace:
I20240503 02:12:47.456142 5159 rpcz_store.cc:276] 0503 02:12:45.472354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.472417 (+ 63us) service_pool.cc:225] Handling call
0503 02:12:47.455872 (+1983455us) ts_tablet_manager.cc:1120] Deleting tablet 92126108432e4450ba30cfde8ca8618f
0503 02:12:47.455883 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.455980 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898396}
W20240503 02:12:47.456503 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet ffef15b7e51840fea91b5349a1724cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffef15b7e51840fea91b5349a1724cc5 already in progress: creating tablet
W20240503 02:12:47.456815 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet db6fa251094842b9a629fdb67e20690b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db6fa251094842b9a629fdb67e20690b already in progress: creating tablet
W20240503 02:12:47.459138 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 92126108432e4450ba30cfde8ca8618f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92126108432e4450ba30cfde8ca8618f already in progress: creating tablet
I20240503 02:12:47.461129 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1039) took 1853 ms. Trace:
I20240503 02:12:47.461251 5159 rpcz_store.cc:276] 0503 02:12:45.607858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.607920 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:47.460962 (+1853042us) ts_tablet_manager.cc:1120] Deleting tablet 1cdfaff4b319415db858f06c47829933
0503 02:12:47.460979 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.461100 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848314}
I20240503 02:12:47.461655 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1038) took 1854 ms. Trace:
I20240503 02:12:47.461733 5159 rpcz_store.cc:276] 0503 02:12:45.606839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.607717 (+ 878us) service_pool.cc:225] Handling call
0503 02:12:47.461520 (+1853803us) ts_tablet_manager.cc:1120] Deleting tablet 368aabce19984d03aa7116c8df990ef1
0503 02:12:47.461532 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.461640 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1820061}
W20240503 02:12:47.461889 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 1cdfaff4b319415db858f06c47829933 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cdfaff4b319415db858f06c47829933 already in progress: creating tablet
I20240503 02:12:47.461990 6905 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:12:47.462064 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1040) took 1589 ms. Trace:
I20240503 02:12:47.462129 5159 rpcz_store.cc:276] 0503 02:12:45.872814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.872871 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:47.461964 (+1589093us) ts_tablet_manager.cc:1120] Deleting tablet 972cd318ebed41c0ab1b8705f52bdaa4
0503 02:12:47.461972 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.462049 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588839}
I20240503 02:12:47.462440 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1053) took 1255 ms. Trace:
I20240503 02:12:47.462512 5159 rpcz_store.cc:276] 0503 02:12:46.207408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.207465 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:47.462356 (+1254891us) ts_tablet_manager.cc:1120] Deleting tablet 620223977a5f48209f6cc12b53915508
0503 02:12:47.462363 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.462425 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254710}
I20240503 02:12:47.462810 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1054) took 1252 ms. Trace:
I20240503 02:12:47.462878 5159 rpcz_store.cc:276] 0503 02:12:46.210838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.210900 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:47.462718 (+1251818us) ts_tablet_manager.cc:1120] Deleting tablet 7a047360ed8843008c4ba633a65b2769
0503 02:12:47.462728 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.462796 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251630}
W20240503 02:12:47.462713 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 368aabce19984d03aa7116c8df990ef1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 368aabce19984d03aa7116c8df990ef1 already in progress: creating tablet
I20240503 02:12:47.463189 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1055) took 1250 ms. Trace:
I20240503 02:12:47.463254 5159 rpcz_store.cc:276] 0503 02:12:46.212441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.212499 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:47.463078 (+1250579us) ts_tablet_manager.cc:1120] Deleting tablet 81213d7b9fc9429abd17f93b9f6982c1
0503 02:12:47.463087 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.463174 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250407}
W20240503 02:12:47.463418 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 972cd318ebed41c0ab1b8705f52bdaa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 972cd318ebed41c0ab1b8705f52bdaa4 already in progress: creating tablet
I20240503 02:12:47.463551 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1056) took 1245 ms. Trace:
W20240503 02:12:47.463627 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 620223977a5f48209f6cc12b53915508 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 620223977a5f48209f6cc12b53915508 already in progress: creating tablet
I20240503 02:12:47.463613 5159 rpcz_store.cc:276] 0503 02:12:46.217714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.217770 (+ 56us) service_pool.cc:225] Handling call
0503 02:12:47.463450 (+1245680us) ts_tablet_manager.cc:1120] Deleting tablet 557f1e3b3df64a8f899b08ba8dccc8f8
0503 02:12:47.463458 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.463538 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245473}
I20240503 02:12:47.463897 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1057) took 1240 ms. Trace:
W20240503 02:12:47.463879 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 7a047360ed8843008c4ba633a65b2769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a047360ed8843008c4ba633a65b2769 already in progress: creating tablet
I20240503 02:12:47.463953 5159 rpcz_store.cc:276] 0503 02:12:46.223062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.223140 (+ 78us) service_pool.cc:225] Handling call
0503 02:12:47.463806 (+1240666us) ts_tablet_manager.cc:1120] Deleting tablet 65ba631d85854b43bcb4bece9bfba7d3
0503 02:12:47.463814 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.463883 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240435}
I20240503 02:12:47.464195 5159 tablet_replica.cc:330] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.464399 5159 raft_consensus.cc:2229] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.464591 5159 raft_consensus.cc:2258] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:47.464649 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 81213d7b9fc9429abd17f93b9f6982c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81213d7b9fc9429abd17f93b9f6982c1 already in progress: creating tablet
W20240503 02:12:47.464872 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 557f1e3b3df64a8f899b08ba8dccc8f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 557f1e3b3df64a8f899b08ba8dccc8f8 already in progress: creating tablet
W20240503 02:12:47.465216 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet 65ba631d85854b43bcb4bece9bfba7d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ba631d85854b43bcb4bece9bfba7d3 already in progress: creating tablet
I20240503 02:12:47.471001 5159 ts_tablet_manager.cc:1875] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.472066 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bf67d8c1d3ec4f758cca9745c42c7ee6 (table default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785]). Expected version 1 got 0
I20240503 02:12:47.396425 6841 raft_consensus.cc:504] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.473038 6948 raft_consensus.cc:482] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.473196 6948 raft_consensus.cc:504] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.473740 6948 leader_election.cc:288] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.473058 6841 leader_election.cc:288] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.478972 4268 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_4a987b64a137496cb458d34c67236398 [id=c99932f82ee449a98a52fbb18dd35129] alter complete (version 1)
I20240503 02:12:47.480113 5106 tablet_bootstrap.cc:492] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.480171 6941 raft_consensus.cc:482] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.480263 5106 ts_tablet_manager.cc:1367] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.000s
I20240503 02:12:47.480317 6941 raft_consensus.cc:504] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.480962 6941 leader_election.cc:288] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.481038 5106 raft_consensus.cc:348] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.481236 5106 raft_consensus.cc:374] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.481287 5106 raft_consensus.cc:729] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.481527 5106 consensus_queue.cc:260] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.481786 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8474c841cb64373a223ad76b43ecf5f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:47.482300 4746 leader_election.cc:341] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.488943 6238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35434 (request call id 1) took 1751 ms. Trace:
I20240503 02:12:47.495167 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b8287b45fa0b492ca82c22164f3a7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.498142 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee34be89bda450296b916163bb10c46" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:47.498358 4810 raft_consensus.cc:3037] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.501178 4810 raft_consensus.cc:2454] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:47.501902 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb174825f1441f861ae6816137f855" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:47.502108 4810 raft_consensus.cc:2454] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:47.502645 6899 raft_consensus.cc:482] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.502797 6899 raft_consensus.cc:504] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.503391 6899 leader_election.cc:288] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:47.504081 4608 leader_election.cc:341] T c757a3a979c2489daa621cf8ea293ea8 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: c757a3a979c2489daa621cf8ea293ea8
I20240503 02:12:47.504369 6899 consensus_queue.cc:1035] T b143c398ccbe42f09fa2e33f04f0d4d7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.504962 4608 leader_election.cc:302] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.505195 6899 raft_consensus.cc:2781] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.505317 6899 raft_consensus.cc:686] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.505532 6899 consensus_queue.cc:237] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:47.507709 4608 leader_election.cc:341] T 94559cfc32704eae96b121f165686edc P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 94559cfc32704eae96b121f165686edc
I20240503 02:12:47.508633 4608 leader_election.cc:302] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.508853 6899 raft_consensus.cc:2781] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.508986 6899 raft_consensus.cc:686] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.509196 6899 consensus_queue.cc:237] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.515432 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_5bb43ae60af64383b89a700d96f6fad7 [id=45c3177a5f154c21a0be1e617e114785] alter complete (version 1)
I20240503 02:12:47.400324 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96c2ce647614957804fc5689a0ee0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.417063 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb174825f1441f861ae6816137f855 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a271c33588f432a801b8cbc0fb79885) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.533365 5106 ts_tablet_manager.cc:1398] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240503 02:12:47.533743 5106 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.533159 4271 catalog_manager.cc:5484] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:47.534214 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc]). Expected version 2 got 0
I20240503 02:12:47.535313 5106 tablet_bootstrap.cc:654] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.536813 4608 leader_election.cc:302] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.537087 6899 raft_consensus.cc:2781] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.537154 6899 raft_consensus.cc:482] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.537214 6899 raft_consensus.cc:3037] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.540268 6899 raft_consensus.cc:504] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.540961 6899 leader_election.cc:288] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.543697 5105 tablet_bootstrap.cc:492] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.543884 5105 ts_tablet_manager.cc:1367] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.020s
I20240503 02:12:47.544651 5105 raft_consensus.cc:348] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.544862 5105 raft_consensus.cc:374] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.544952 5105 raft_consensus.cc:729] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.545172 5105 consensus_queue.cc:260] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.545655 5105 ts_tablet_manager.cc:1398] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:47.548329 5105 tablet_bootstrap.cc:492] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.549757 5105 tablet_bootstrap.cc:654] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.555732 4640 tablet_service.cc:1467] Processing CreateTablet for tablet c70d86c941f2445fbd6ff8b0be7e4043 (DEFAULT_TABLE table=default.Table_4511fe14eef348aa8f91987405ac7bfb [id=562a3fed23794d9d8ca6ae700b979883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.556432 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet d607da9689c14e51add9178a48b35913 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.557861 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb174825f1441f861ae6816137f855" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:47.558094 4805 raft_consensus.cc:3037] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.562491 4805 raft_consensus.cc:2454] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:47.563563 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet c3710b653aa24b7ca389aa95d7cdd2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.570312 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70d86c941f2445fbd6ff8b0be7e4043. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:47.570386 4608 consensus_peers.cc:485] T ff8d2c0991e849b5afa35e7230db298d P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8d2c0991e849b5afa35e7230db298d. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:47.570556 4608 leader_election.cc:341] T 0cc2ebfb11aa4840adcc35da73be75a2 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 0cc2ebfb11aa4840adcc35da73be75a2
I20240503 02:12:47.570749 4608 leader_election.cc:302] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.571137 6899 raft_consensus.cc:2781] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.571349 6899 raft_consensus.cc:686] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:47.571568 6899 consensus_queue.cc:237] T fcfb174825f1441f861ae6816137f855 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.574483 5105 tablet_bootstrap.cc:492] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.574658 5105 ts_tablet_manager.cc:1367] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20240503 02:12:47.575452 5105 raft_consensus.cc:348] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.575682 5105 raft_consensus.cc:374] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.575737 5105 raft_consensus.cc:729] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.575991 5105 consensus_queue.cc:260] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.576740 5115 ts_tablet_manager.cc:1888] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.576764 5105 ts_tablet_manager.cc:1398] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:47.576974 5115 log.cc:1196] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/4b4972057fae45168756d7c28ddbe2bf
I20240503 02:12:47.577180 5105 tablet_bootstrap.cc:492] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.577584 5115 ts_tablet_manager.cc:1909] T 4b4972057fae45168756d7c28ddbe2bf P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.578153 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 876) took 5185 ms. Trace:
I20240503 02:12:47.578325 5115 rpcz_store.cc:276] 0503 02:12:42.392549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.392938 (+ 389us) service_pool.cc:225] Handling call
0503 02:12:47.382561 (+4989623us) ts_tablet_manager.cc:1120] Deleting tablet 4b4972057fae45168756d7c28ddbe2bf
0503 02:12:47.382573 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.569800 (+187227us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.576622 (+ 6822us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.578128 (+ 1506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4989488}
I20240503 02:12:47.578846 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 877) took 5111 ms. Trace:
I20240503 02:12:47.578984 5115 rpcz_store.cc:276] 0503 02:12:42.466906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.466975 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:47.578707 (+5111732us) ts_tablet_manager.cc:1120] Deleting tablet 5a0303f1b1194115a7142a00470457c3
0503 02:12:47.578718 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.578830 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111498}
I20240503 02:12:47.579226 5105 tablet_bootstrap.cc:654] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.580621 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 4b4972057fae45168756d7c28ddbe2bf (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]) successfully deleted
I20240503 02:12:47.583034 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 878) took 5112 ms. Trace:
I20240503 02:12:47.583232 5115 rpcz_store.cc:276] 0503 02:12:42.470506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:42.470567 (+ 61us) service_pool.cc:225] Handling call
0503 02:12:47.582911 (+5112344us) ts_tablet_manager.cc:1120] Deleting tablet 01c61bf67c8a47fb9ef23bcd6c58c448
0503 02:12:47.582919 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.583016 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5112168}
I20240503 02:12:47.583699 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 892) took 4541 ms. Trace:
I20240503 02:12:47.583823 5115 rpcz_store.cc:276] 0503 02:12:43.041978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.042020 (+ 42us) service_pool.cc:225] Handling call
0503 02:12:47.583590 (+4541570us) ts_tablet_manager.cc:1120] Deleting tablet b5f444b22112407ca02fe86ac50167a9
0503 02:12:47.583599 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.583684 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4541399}
I20240503 02:12:47.584195 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 893) took 4541 ms. Trace:
I20240503 02:12:47.584335 5115 rpcz_store.cc:276] 0503 02:12:43.042301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.042338 (+ 37us) service_pool.cc:225] Handling call
0503 02:12:47.584101 (+4541763us) ts_tablet_manager.cc:1120] Deleting tablet a6de1b96c15c4b44809a03ef99e077ba
0503 02:12:47.584109 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.584181 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4541629}
I20240503 02:12:47.585012 5105 tablet_bootstrap.cc:492] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:47.585197 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 5a0303f1b1194115a7142a00470457c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a0303f1b1194115a7142a00470457c3 already in progress: creating tablet
I20240503 02:12:47.585264 5105 ts_tablet_manager.cc:1367] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
W20240503 02:12:47.585896 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 01c61bf67c8a47fb9ef23bcd6c58c448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c61bf67c8a47fb9ef23bcd6c58c448 already in progress: creating tablet
I20240503 02:12:47.585975 5105 raft_consensus.cc:348] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.586190 5105 raft_consensus.cc:374] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.586249 5105 raft_consensus.cc:729] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:47.586396 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b5f444b22112407ca02fe86ac50167a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f444b22112407ca02fe86ac50167a9 already in progress: creating tablet
I20240503 02:12:47.586491 5105 consensus_queue.cc:260] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.587023 5105 ts_tablet_manager.cc:1398] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 02:12:47.587011 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet a6de1b96c15c4b44809a03ef99e077ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6de1b96c15c4b44809a03ef99e077ba already in progress: creating tablet
I20240503 02:12:47.587321 5105 tablet_bootstrap.cc:492] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.587280 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 891) took 4545 ms. Trace:
I20240503 02:12:47.587369 5115 rpcz_store.cc:276] 0503 02:12:43.041911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.042410 (+ 499us) service_pool.cc:225] Handling call
0503 02:12:47.584603 (+4542193us) ts_tablet_manager.cc:1120] Deleting tablet b2560f50415444b8b2e497453a558d3a
0503 02:12:47.584666 (+ 63us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.587262 (+ 2596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4542036}
I20240503 02:12:47.587680 5115 tablet_replica.cc:330] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.587877 5115 raft_consensus.cc:2229] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:12:47.587958 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet b2560f50415444b8b2e497453a558d3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2560f50415444b8b2e497453a558d3a already in progress: creating tablet
I20240503 02:12:47.588198 5115 raft_consensus.cc:2258] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.588867 5115 ts_tablet_manager.cc:1875] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.593402 5105 tablet_bootstrap.cc:654] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.600648 5115 ts_tablet_manager.cc:1888] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.600986 5115 log.cc:1196] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/cd66f88a24b74f2c9b2050af081d4601
I20240503 02:12:47.602416 6941 raft_consensus.cc:482] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.602571 6941 raft_consensus.cc:504] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.603404 6941 leader_election.cc:288] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.604138 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54959b3e30944fab8bd5fd47ce42d246" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.607625 5115 ts_tablet_manager.cc:1909] T cd66f88a24b74f2c9b2050af081d4601 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:47.607728 4746 leader_election.cc:341] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.608937 5105 tablet_bootstrap.cc:492] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.608955 5106 tablet_bootstrap.cc:492] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.609073 5105 ts_tablet_manager.cc:1367] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240503 02:12:47.609107 5106 ts_tablet_manager.cc:1367] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240503 02:12:47.609843 5105 raft_consensus.cc:348] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.610056 5105 raft_consensus.cc:374] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.610110 5105 raft_consensus.cc:729] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.610559 5105 consensus_queue.cc:260] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.611083 5105 ts_tablet_manager.cc:1398] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:47.611361 5105 tablet_bootstrap.cc:492] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.612857 5105 tablet_bootstrap.cc:654] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.611022 5106 raft_consensus.cc:348] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.613202 5106 raft_consensus.cc:374] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.613313 5106 raft_consensus.cc:729] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.613314 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:47.613612 5106 consensus_queue.cc:260] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.614971 5106 ts_tablet_manager.cc:1398] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:47.618728 5105 tablet_bootstrap.cc:492] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.618902 5105 ts_tablet_manager.cc:1367] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240503 02:12:47.619742 5105 raft_consensus.cc:348] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.619962 5105 raft_consensus.cc:374] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.620024 5105 raft_consensus.cc:729] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.620242 5105 consensus_queue.cc:260] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.620956 5105 ts_tablet_manager.cc:1398] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:47.624343 5105 tablet_bootstrap.cc:492] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.625443 5106 tablet_bootstrap.cc:492] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.627897 5106 tablet_bootstrap.cc:654] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.630504 5105 tablet_bootstrap.cc:654] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.631031 4267 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet a73ec6543b9a468492f3b97772ce43f9 (table default.Table_2aec4967b23a414596b8b354cf8de2d9 [id=eb724fd2c11544ab9d8d539d30a2ffa4]). Expected version 2 got 0
I20240503 02:12:47.629487 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 894) took 4586 ms. Trace:
I20240503 02:12:47.633194 5115 rpcz_store.cc:276] 0503 02:12:43.042570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.042711 (+ 141us) service_pool.cc:225] Handling call
0503 02:12:47.587636 (+4544925us) ts_tablet_manager.cc:1120] Deleting tablet cd66f88a24b74f2c9b2050af081d4601
0503 02:12:47.587645 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.597124 (+ 9479us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.600286 (+ 3162us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.629460 (+ 29174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4544781}
I20240503 02:12:47.628448 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet a86ac17982324847949b8a9245329489 (table default.Table_a970660694dc4ff5b2a356e5d36d2beb [id=4dbb0ce32dad4373a8362c9297990767]). Expected version 2 got 0
I20240503 02:12:47.634168 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 890fb92ca2314f2e81e1ffb0bcddb0b1 (table default.Table_9d93e9e41b354e29a8f41a5dfdf3a32a [id=1b3b3d86efe94e1d8f6230f7ceb285cd]). Expected version 1 got 0
I20240503 02:12:47.636037 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet cd66f88a24b74f2c9b2050af081d4601 (table default.Table_3d65228f6c2b4694bb60f3118ad65af7 [id=2d5bfb965c844a199fa81636be8265a6]) successfully deleted
I20240503 02:12:47.639967 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 with delete_type TABLET_DATA_DELETED (replaced by tablet b72bff3c50eb4ac6b1b2feb7250ba43e) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.651371 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:47.653117 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b6a18be64049049e1a083bf4af51ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.656405 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 886) took 4629 ms. Trace:
I20240503 02:12:47.657945 5115 rpcz_store.cc:276] 0503 02:12:43.027406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.087325 (+ 59919us) service_pool.cc:225] Handling call
0503 02:12:47.633565 (+4546240us) ts_tablet_manager.cc:1120] Deleting tablet 9e4dabd6e05948b98f02515023e2aaa1
0503 02:12:47.656153 (+ 22588us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.656361 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545628}
I20240503 02:12:47.659411 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 901) took 4443 ms. Trace:
W20240503 02:12:47.659475 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9e4dabd6e05948b98f02515023e2aaa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e4dabd6e05948b98f02515023e2aaa1 already in progress: creating tablet
I20240503 02:12:47.660542 5115 rpcz_store.cc:276] 0503 02:12:43.215448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.232532 (+ 17084us) service_pool.cc:225] Handling call
0503 02:12:47.659211 (+4426679us) ts_tablet_manager.cc:1120] Deleting tablet 192460518d49417cb8f03ac6ec971eef
0503 02:12:47.659229 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.659383 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4426448}
I20240503 02:12:47.658618 6830 raft_consensus.cc:482] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.661258 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 904) took 4261 ms. Trace:
I20240503 02:12:47.661088 6830 raft_consensus.cc:504] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.661921 6830 leader_election.cc:288] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.661366 5115 rpcz_store.cc:276] 0503 02:12:43.399430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.399494 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:47.661068 (+4261574us) ts_tablet_manager.cc:1120] Deleting tablet cbaa94834cdc44eabe71242a698e3639
0503 02:12:47.661083 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.661235 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4261347}
I20240503 02:12:47.662580 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 905) took 4263 ms. Trace:
I20240503 02:12:47.662685 5115 rpcz_store.cc:276] 0503 02:12:43.399584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.399745 (+ 161us) service_pool.cc:225] Handling call
0503 02:12:47.662470 (+4262725us) ts_tablet_manager.cc:1120] Deleting tablet f8be9e5380b4466baccd16483e940588
0503 02:12:47.662477 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.662563 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4262576}
W20240503 02:12:47.662721 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 192460518d49417cb8f03ac6ec971eef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 192460518d49417cb8f03ac6ec971eef already in progress: creating tablet
W20240503 02:12:47.663116 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f8be9e5380b4466baccd16483e940588 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8be9e5380b4466baccd16483e940588 already in progress: creating tablet
W20240503 02:12:47.666883 4608 leader_election.cc:341] T ee1fac803dd84eee8722f0a14559faf6 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: ee1fac803dd84eee8722f0a14559faf6
I20240503 02:12:47.670266 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet a73ec6543b9a468492f3b97772ce43f9 (table default.Table_2aec4967b23a414596b8b354cf8de2d9 [id=eb724fd2c11544ab9d8d539d30a2ffa4]). Expected version 2 got 0
I20240503 02:12:47.676303 6941 raft_consensus.cc:482] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.676401 5159 ts_tablet_manager.cc:1888] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.676549 5159 log.cc:1196] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/aa67857b858c4719b498291c52b7aceb
I20240503 02:12:47.676513 6941 raft_consensus.cc:504] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.677201 6941 leader_election.cc:288] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.678136 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11c306232c4e49ab02be0f195df935" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:47.678866 4746 leader_election.cc:341] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.679709 5115 tablet_replica.cc:330] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.680027 5115 raft_consensus.cc:2229] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.680274 5115 raft_consensus.cc:2258] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.682204 5115 ts_tablet_manager.cc:1875] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.685482 5159 ts_tablet_manager.cc:1909] T aa67857b858c4719b498291c52b7aceb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.686069 5106 tablet_bootstrap.cc:492] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.686280 5106 ts_tablet_manager.cc:1367] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.004s
I20240503 02:12:47.686476 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1074) took 1134 ms. Trace:
I20240503 02:12:47.686630 5159 rpcz_store.cc:276] 0503 02:12:46.551884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.551962 (+ 78us) service_pool.cc:225] Handling call
0503 02:12:47.464139 (+912177us) ts_tablet_manager.cc:1120] Deleting tablet aa67857b858c4719b498291c52b7aceb
0503 02:12:47.464158 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.565915 (+101757us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.568871 (+ 2956us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.686451 (+117580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":911953}
I20240503 02:12:47.687160 5106 raft_consensus.cc:348] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.687317 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet aa67857b858c4719b498291c52b7aceb (table default.Table_ceabc3a7225649c198519f5a34379f63 [id=5c72e153c1034228b769b20591c2d596]) successfully deleted
I20240503 02:12:47.687474 5106 raft_consensus.cc:374] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.687536 5106 raft_consensus.cc:729] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.687791 5159 tablet_replica.cc:330] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.687809 5106 consensus_queue.cc:260] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.688045 5159 raft_consensus.cc:2229] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.688387 5159 raft_consensus.cc:2258] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.688465 5106 ts_tablet_manager.cc:1398] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:47.688800 5106 tablet_bootstrap.cc:492] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.689131 5159 ts_tablet_manager.cc:1875] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.690521 5106 tablet_bootstrap.cc:654] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.692710 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet b143c398ccbe42f09fa2e33f04f0d4d7 (table default.Table_dddd71becaf8450f999400d322914922 [id=db26d21e021444d795aade9f022af228]). Expected version 6 got 0
I20240503 02:12:47.717864 5115 ts_tablet_manager.cc:1888] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.718252 5115 log.cc:1196] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/035216c9475d4b59aa1ac81b285b4d8a
I20240503 02:12:47.718945 5115 ts_tablet_manager.cc:1909] T 035216c9475d4b59aa1ac81b285b4d8a P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.720192 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 908) took 4152 ms. Trace:
I20240503 02:12:47.720438 5115 rpcz_store.cc:276] 0503 02:12:43.568090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.568374 (+ 284us) service_pool.cc:225] Handling call
0503 02:12:47.662922 (+4094548us) ts_tablet_manager.cc:1120] Deleting tablet 035216c9475d4b59aa1ac81b285b4d8a
0503 02:12:47.663038 (+ 116us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.712058 (+ 49020us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.717695 (+ 5637us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.720147 (+ 2452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4094342}
I20240503 02:12:47.721269 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 035216c9475d4b59aa1ac81b285b4d8a (table default.Table_1ea22186c7c9467c8a58b89fdf817b19 [id=2b81e59218084f5c9aa8611c82c0c54a]) successfully deleted
I20240503 02:12:47.721918 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 909) took 4153 ms. Trace:
I20240503 02:12:47.722100 5115 rpcz_store.cc:276] 0503 02:12:43.568200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.568618 (+ 418us) service_pool.cc:225] Handling call
0503 02:12:47.721771 (+4153153us) ts_tablet_manager.cc:1120] Deleting tablet 7461aa76a1af41cb9253285bda708f30
0503 02:12:47.721784 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.721894 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4152997}
W20240503 02:12:47.723086 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7461aa76a1af41cb9253285bda708f30 because the tablet was not found. No further retry: Not found: Tablet not found: 7461aa76a1af41cb9253285bda708f30
I20240503 02:12:47.723728 5115 tablet_replica.cc:330] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.724130 5115 raft_consensus.cc:2229] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.724450 5115 raft_consensus.cc:2258] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.725576 5115 ts_tablet_manager.cc:1875] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.726930 5106 tablet_bootstrap.cc:492] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.727154 5106 ts_tablet_manager.cc:1367] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240503 02:12:47.727996 5106 raft_consensus.cc:348] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.728327 5106 raft_consensus.cc:374] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.728428 5106 raft_consensus.cc:729] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.728720 5106 consensus_queue.cc:260] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.731353 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aad7d6dcf2e4009872ce156c536f38e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.735535 5106 ts_tablet_manager.cc:1398] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:12:47.735982 5106 tablet_bootstrap.cc:492] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.737845 5106 tablet_bootstrap.cc:654] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.746588 5106 tablet_bootstrap.cc:492] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.746765 5106 ts_tablet_manager.cc:1367] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 02:12:47.747550 5106 raft_consensus.cc:348] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.747815 5106 raft_consensus.cc:374] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.747890 5106 raft_consensus.cc:729] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.748111 5106 consensus_queue.cc:260] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.748646 5106 ts_tablet_manager.cc:1398] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:47.748946 5106 tablet_bootstrap.cc:492] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.751493 5115 ts_tablet_manager.cc:1888] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.751770 5106 tablet_bootstrap.cc:654] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.763605 5105 tablet_bootstrap.cc:492] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.763767 5105 ts_tablet_manager.cc:1367] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.140s user 0.062s sys 0.058s
I20240503 02:12:47.764511 5105 raft_consensus.cc:348] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.764729 5105 raft_consensus.cc:374] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.764784 5105 raft_consensus.cc:729] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.765022 5105 consensus_queue.cc:260] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.765640 5105 ts_tablet_manager.cc:1398] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:47.765946 5105 tablet_bootstrap.cc:492] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.767140 5115 log.cc:1196] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/43119471f91c4a1ba900f267c88db9a6
I20240503 02:12:47.767372 5105 tablet_bootstrap.cc:654] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.767580 5115 ts_tablet_manager.cc:1909] T 43119471f91c4a1ba900f267c88db9a6 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.768087 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 910) took 4199 ms. Trace:
I20240503 02:12:47.768183 5115 rpcz_store.cc:276] 0503 02:12:43.568316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.568800 (+ 484us) service_pool.cc:225] Handling call
0503 02:12:47.723650 (+4154850us) ts_tablet_manager.cc:1120] Deleting tablet 43119471f91c4a1ba900f267c88db9a6
0503 02:12:47.723666 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.747922 (+ 24256us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.751360 (+ 3438us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.768062 (+ 16702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4154716}
I20240503 02:12:47.768662 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 912) took 4149 ms. Trace:
I20240503 02:12:47.768743 5115 rpcz_store.cc:276] 0503 02:12:43.619656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.621845 (+ 2189us) service_pool.cc:225] Handling call
0503 02:12:47.768497 (+4146652us) ts_tablet_manager.cc:1120] Deleting tablet 0794df2bfc5e45db95cf4c745892e695
0503 02:12:47.768508 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.768642 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4146481}
I20240503 02:12:47.769054 5115 tablet_replica.cc:330] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.769232 5115 raft_consensus.cc:2229] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.769526 5115 raft_consensus.cc:2258] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.770082 5115 ts_tablet_manager.cc:1875] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.780115 5159 ts_tablet_manager.cc:1888] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.780391 5159 log.cc:1196] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/dee34be89bda450296b916163bb10c46
I20240503 02:12:47.780855 5159 ts_tablet_manager.cc:1909] T dee34be89bda450296b916163bb10c46 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.783736 5115 ts_tablet_manager.cc:1888] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.783901 5115 log.cc:1196] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/88b173d78ba74ede801b12cccbeef6b1
I20240503 02:12:47.784260 5115 ts_tablet_manager.cc:1909] T 88b173d78ba74ede801b12cccbeef6b1 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.789136 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 914) took 4159 ms. Trace:
I20240503 02:12:47.789263 5115 rpcz_store.cc:276] 0503 02:12:43.630120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.630182 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:47.769008 (+4138826us) ts_tablet_manager.cc:1120] Deleting tablet 88b173d78ba74ede801b12cccbeef6b1
0503 02:12:47.769016 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.773246 (+ 4230us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.783608 (+ 10362us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.789106 (+ 5498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4138630}
I20240503 02:12:47.792526 5115 tablet_replica.cc:330] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.793115 5115 raft_consensus.cc:2229] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.793809 5115 raft_consensus.cc:2258] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.794879 5115 ts_tablet_manager.cc:1875] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:47.771785 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 0794df2bfc5e45db95cf4c745892e695 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0794df2bfc5e45db95cf4c745892e695 already in progress: creating tablet
I20240503 02:12:47.802253 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 88b173d78ba74ede801b12cccbeef6b1 (table default.Table_e6aa097db3d14a28992284c5876aba0f [id=4ac2b5ac834b403980d0cba45d2dcf99]) successfully deleted
I20240503 02:12:47.806766 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_9091ef41bcba4103811266c2b3782904 [id=fde711a2c8e04377996a7cd87f629e68]). Expected version 2 got 0
I20240503 02:12:47.810534 5106 tablet_bootstrap.cc:492] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.810708 5106 ts_tablet_manager.cc:1367] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.019s
I20240503 02:12:47.811374 5106 raft_consensus.cc:348] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.811585 5106 raft_consensus.cc:374] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.811672 5106 raft_consensus.cc:729] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.811882 5106 consensus_queue.cc:260] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.812541 5106 ts_tablet_manager.cc:1398] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:47.816066 5115 ts_tablet_manager.cc:1888] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.816510 5106 tablet_bootstrap.cc:492] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.818589 5106 tablet_bootstrap.cc:654] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.820070 5115 log.cc:1196] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/79b8db679ba24ac6a93be591c0513ab0
W20240503 02:12:47.820214 4608 consensus_peers.cc:485] T 367de4ff0a6f4abfa2924a070f387cbd P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367de4ff0a6f4abfa2924a070f387cbd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:47.820585 5115 ts_tablet_manager.cc:1909] T 79b8db679ba24ac6a93be591c0513ab0 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.786598 6941 raft_consensus.cc:482] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.796079 6814 raft_consensus.cc:482] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.803056 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e1c2fa9cb3481dafb9d7f1e826a30c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:47.821076 6941 raft_consensus.cc:504] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.821846 6941 leader_election.cc:288] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.822074 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1075) took 1260 ms. Trace:
I20240503 02:12:47.821096 6814 raft_consensus.cc:504] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.822362 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce9d746acfe4e9aaea600293470d373" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.822906 6814 leader_election.cc:288] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:47.823014 4746 leader_election.cc:341] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.823500 6814 raft_consensus.cc:482] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.823570 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd296bb3b574d6cbb0252ad10ada00c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.823621 6814 raft_consensus.cc:504] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:47.824012 4746 leader_election.cc:341] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.824147 6814 leader_election.cc:288] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.824601 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32c377006a24372859ed44d59cf90ee" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.824786 4672 raft_consensus.cc:2454] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:47.825201 4746 leader_election.cc:302] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.825765 6814 raft_consensus.cc:2781] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.825845 6814 raft_consensus.cc:482] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.825906 6814 raft_consensus.cc:3037] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.828711 6814 raft_consensus.cc:504] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.830263 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32c377006a24372859ed44d59cf90ee" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:47.830480 4672 raft_consensus.cc:3037] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.833501 4672 raft_consensus.cc:2454] T b32c377006a24372859ed44d59cf90ee P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:47.834100 4746 leader_election.cc:302] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.834532 6814 leader_election.cc:288] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.834736 6814 raft_consensus.cc:2781] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.834901 6814 raft_consensus.cc:686] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:47.835222 6814 consensus_queue.cc:237] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.837698 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_9091ef41bcba4103811266c2b3782904 [id=fde711a2c8e04377996a7cd87f629e68]). Expected version 2 got 0
I20240503 02:12:47.822191 5159 rpcz_store.cc:276] 0503 02:12:46.561300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.561379 (+ 79us) service_pool.cc:225] Handling call
0503 02:12:47.687729 (+1126350us) ts_tablet_manager.cc:1120] Deleting tablet dee34be89bda450296b916163bb10c46
0503 02:12:47.687742 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.725810 (+ 38068us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.779928 (+ 54118us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.822042 (+ 42114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126086}
I20240503 02:12:47.838804 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 54959b3e30944fab8bd5fd47ce42d246 (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]). Expected version 2 got 0
I20240503 02:12:47.840651 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet dee34be89bda450296b916163bb10c46 (table default.Table_23526249cb074ae5b90c63d59d47e828 [id=9f241c41ddaf480fa60902a73a61c36f]) successfully deleted
I20240503 02:12:47.841341 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1076) took 1277 ms. Trace:
I20240503 02:12:47.841753 5159 rpcz_store.cc:276] 0503 02:12:46.564102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.564178 (+ 76us) service_pool.cc:225] Handling call
0503 02:12:47.841165 (+1276987us) ts_tablet_manager.cc:1120] Deleting tablet e4cb6432296c4a45850a96c3651ab7bd
0503 02:12:47.841179 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.841317 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249295}
I20240503 02:12:47.842360 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 915) took 4211 ms. Trace:
W20240503 02:12:47.842916 4259 catalog_manager.cc:4810] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): delete failed for tablet e4cb6432296c4a45850a96c3651ab7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cb6432296c4a45850a96c3651ab7bd already in progress: creating tablet
I20240503 02:12:47.843263 5115 rpcz_store.cc:276] 0503 02:12:43.630504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.630549 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:47.792465 (+4161916us) ts_tablet_manager.cc:1120] Deleting tablet 79b8db679ba24ac6a93be591c0513ab0
0503 02:12:47.792477 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.812968 (+ 20491us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.815951 (+ 2983us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.842336 (+ 26385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4161744}
I20240503 02:12:47.843663 5115 tablet_replica.cc:330] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.843875 5115 raft_consensus.cc:2229] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.843901 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 79b8db679ba24ac6a93be591c0513ab0 (table default.Table_95c5dbfc7e2542dd93ea5839f66200e1 [id=ffbb2c8938d54556bf18d766e0c901cc]) successfully deleted
I20240503 02:12:47.846383 5115 raft_consensus.cc:2258] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.847507 5115 ts_tablet_manager.cc:1875] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.857693 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet 3478033c4ed94268a2436a2942c27a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:28 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:47.843451 5159 tablet_replica.cc:330] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.859916 5106 tablet_bootstrap.cc:492] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.860152 5106 ts_tablet_manager.cc:1367] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.002s
I20240503 02:12:47.861004 5106 raft_consensus.cc:348] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.861310 5106 raft_consensus.cc:374] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.861416 5106 raft_consensus.cc:729] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.861717 5106 consensus_queue.cc:260] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.867761 5106 ts_tablet_manager.cc:1398] T a5f8d331a718475a9f56dc7f3ec0e6b9 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:47.868201 5106 tablet_bootstrap.cc:492] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.870240 5106 tablet_bootstrap.cc:654] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.873173 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (table default.Table_ec81cfea38bf4719b2e30c82d4d93f50 [id=997231149d3e427f824ed751b345d7b8]). Expected version 2 got 0
I20240503 02:12:47.877457 5115 ts_tablet_manager.cc:1888] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.877599 5115 log.cc:1196] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/b38d63d2cc8c4f54ba43cdf8cde1841f
I20240503 02:12:47.877983 5115 ts_tablet_manager.cc:1909] T b38d63d2cc8c4f54ba43cdf8cde1841f P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.878823 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 916) took 4248 ms. Trace:
I20240503 02:12:47.878926 5115 rpcz_store.cc:276] 0503 02:12:43.630732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.630773 (+ 41us) service_pool.cc:225] Handling call
0503 02:12:47.843604 (+4212831us) ts_tablet_manager.cc:1120] Deleting tablet b38d63d2cc8c4f54ba43cdf8cde1841f
0503 02:12:47.843618 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.857734 (+ 14116us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.877343 (+ 19609us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.878801 (+ 1458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4212639}
I20240503 02:12:47.879319 5115 tablet_replica.cc:330] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.879514 5115 raft_consensus.cc:2229] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.879828 5115 pending_rounds.cc:62] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Trying to abort 1 pending ops.
I20240503 02:12:47.879885 5115 pending_rounds.cc:68] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420881035038720 op_type: NO_OP noop_request { }
I20240503 02:12:47.880000 5115 raft_consensus.cc:2866] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:12:47.880084 5115 raft_consensus.cc:2258] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.880674 5115 ts_tablet_manager.cc:1875] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.881078 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet b38d63d2cc8c4f54ba43cdf8cde1841f (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]) successfully deleted
I20240503 02:12:47.889041 6830 raft_consensus.cc:482] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:47.889304 6830 raft_consensus.cc:504] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.890305 6830 leader_election.cc:288] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.890895 5115 ts_tablet_manager.cc:1888] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.891036 5115 log.cc:1196] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/8195767d80384a6c9a4690292f6776de
W20240503 02:12:47.891980 4608 leader_election.cc:341] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 2f862891b43a4845b723bfdb9a7da591
I20240503 02:12:47.894912 5159 raft_consensus.cc:2229] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.895220 5159 raft_consensus.cc:2258] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.896044 5159 ts_tablet_manager.cc:1875] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.896368 5115 ts_tablet_manager.cc:1909] T 8195767d80384a6c9a4690292f6776de P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.902082 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet a5f8d331a718475a9f56dc7f3ec0e6b9 (table default.Table_ec81cfea38bf4719b2e30c82d4d93f50 [id=997231149d3e427f824ed751b345d7b8]). Expected version 2 got 0
I20240503 02:12:47.905772 5105 tablet_bootstrap.cc:492] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.905941 5105 ts_tablet_manager.cc:1367] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.140s user 0.056s sys 0.079s
I20240503 02:12:47.906726 5105 raft_consensus.cc:348] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.906945 5105 raft_consensus.cc:374] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.907008 5105 raft_consensus.cc:729] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.907213 5105 consensus_queue.cc:260] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.907799 5105 ts_tablet_manager.cc:1398] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:47.908089 5105 tablet_bootstrap.cc:492] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.903337 4271 catalog_manager.cc:5484] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:47.913571 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet b32c377006a24372859ed44d59cf90ee (table default.Table_80dedb822df54ccc9e63c27319fec3b9 [id=921e31bf406040b9a44409fdfbb79d2e]). Expected version 2 got 0
I20240503 02:12:47.914544 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 913) took 4284 ms. Trace:
I20240503 02:12:47.914675 5115 rpcz_store.cc:276] 0503 02:12:43.629864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:43.632963 (+ 3099us) service_pool.cc:225] Handling call
0503 02:12:47.879250 (+4246287us) ts_tablet_manager.cc:1120] Deleting tablet 8195767d80384a6c9a4690292f6776de
0503 02:12:47.879259 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.887575 (+ 8316us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.890773 (+ 3198us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.914514 (+ 23741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4246074}
I20240503 02:12:47.915169 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 934) took 3892 ms. Trace:
I20240503 02:12:47.915269 5115 rpcz_store.cc:276] 0503 02:12:44.023087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.023145 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:47.915019 (+3891874us) ts_tablet_manager.cc:1120] Deleting tablet 151f925dcb304594b60c85d850b83533
0503 02:12:47.915031 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.915151 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3891624}
I20240503 02:12:47.915596 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 935) took 3880 ms. Trace:
I20240503 02:12:47.915663 5115 rpcz_store.cc:276] 0503 02:12:44.035174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.035239 (+ 65us) service_pool.cc:225] Handling call
0503 02:12:47.915487 (+3880248us) ts_tablet_manager.cc:1120] Deleting tablet e82190dfb1b1418b9f20bf4d0c2c2409
0503 02:12:47.915497 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.915580 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3879926}
I20240503 02:12:47.915972 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 936) took 3879 ms. Trace:
I20240503 02:12:47.916041 5115 rpcz_store.cc:276] 0503 02:12:44.036480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.036557 (+ 77us) service_pool.cc:225] Handling call
0503 02:12:47.915875 (+3879318us) ts_tablet_manager.cc:1120] Deleting tablet 3ee32345fb9349f19af09a3447c4caad
0503 02:12:47.915883 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.915959 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3879066}
I20240503 02:12:47.916371 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 937) took 3877 ms. Trace:
I20240503 02:12:47.916411 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 8195767d80384a6c9a4690292f6776de (table default.Table_c9a280da9b284233bbd84cc8bb291cca [id=77739df395834c17a23912a773ff5c1d]) successfully deleted
I20240503 02:12:47.916445 5115 rpcz_store.cc:276] 0503 02:12:44.038583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.038633 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:47.916281 (+3877648us) ts_tablet_manager.cc:1120] Deleting tablet 2848c837598c49628c11374d3c875b3e
0503 02:12:47.916289 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.916357 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874638}
I20240503 02:12:47.916690 5115 tablet_replica.cc:330] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:47.916685 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 151f925dcb304594b60c85d850b83533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 151f925dcb304594b60c85d850b83533 already in progress: creating tablet
I20240503 02:12:47.916868 5115 raft_consensus.cc:2229] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:12:47.916906 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet e82190dfb1b1418b9f20bf4d0c2c2409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82190dfb1b1418b9f20bf4d0c2c2409 already in progress: creating tablet
I20240503 02:12:47.917040 5115 raft_consensus.cc:2258] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:47.917376 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 3ee32345fb9349f19af09a3447c4caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ee32345fb9349f19af09a3447c4caad already in progress: creating tablet
W20240503 02:12:47.917596 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 2848c837598c49628c11374d3c875b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2848c837598c49628c11374d3c875b3e already in progress: creating tablet
I20240503 02:12:47.917594 5115 ts_tablet_manager.cc:1875] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.918723 6830 raft_consensus.cc:482] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.918897 6830 raft_consensus.cc:504] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.918998 5159 ts_tablet_manager.cc:1888] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.919131 5159 log.cc:1196] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3aad7d6dcf2e4009872ce156c536f38e
I20240503 02:12:47.919610 6830 leader_election.cc:288] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.920132 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aad7d6dcf2e4009872ce156c536f38e" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
W20240503 02:12:47.921281 4608 leader_election.cc:341] T 3aad7d6dcf2e4009872ce156c536f38e P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: STOPPED
I20240503 02:12:47.921464 5159 ts_tablet_manager.cc:1909] T 3aad7d6dcf2e4009872ce156c536f38e P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.922288 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1077) took 1039 ms. Trace:
I20240503 02:12:47.922380 5159 rpcz_store.cc:276] 0503 02:12:46.882960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:46.883017 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:47.843381 (+960364us) ts_tablet_manager.cc:1120] Deleting tablet 3aad7d6dcf2e4009872ce156c536f38e
0503 02:12:47.843394 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.910567 (+ 67173us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.918877 (+ 8310us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.922266 (+ 3389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7449,"tablet-delete.queue_time_us":953293}
I20240503 02:12:47.922824 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3aad7d6dcf2e4009872ce156c536f38e (table default.Table_4489651bcc554f9ca32e9f7a9629b9ce [id=987ed61093524601ba8886c4d508565a]) successfully deleted
I20240503 02:12:47.922886 5159 tablet_replica.cc:330] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.923070 5159 raft_consensus.cc:2229] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.923264 5159 raft_consensus.cc:2258] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.923877 5159 ts_tablet_manager.cc:1875] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.924227 5115 ts_tablet_manager.cc:1888] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.924357 5115 log.cc:1196] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/81d6d3ca80c54c629a5de82b06a7aac4
I20240503 02:12:47.924711 5115 ts_tablet_manager.cc:1909] T 81d6d3ca80c54c629a5de82b06a7aac4 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.925606 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 939) took 3881 ms. Trace:
I20240503 02:12:47.925702 5115 rpcz_store.cc:276] 0503 02:12:44.044127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.044186 (+ 59us) service_pool.cc:225] Handling call
0503 02:12:47.916637 (+3872451us) ts_tablet_manager.cc:1120] Deleting tablet 81d6d3ca80c54c629a5de82b06a7aac4
0503 02:12:47.916645 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.920739 (+ 4094us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.924055 (+ 3316us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.925583 (+ 1528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3872288}
I20240503 02:12:47.926076 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 81d6d3ca80c54c629a5de82b06a7aac4 (table default.Table_27e6472c5d7746f59ee6399fa6db4d3b [id=74533d0ffe7340578b7b2201d27f7411]) successfully deleted
I20240503 02:12:47.926218 5115 tablet_replica.cc:330] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.926415 5115 raft_consensus.cc:2229] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:47.926697 5115 raft_consensus.cc:2258] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.927222 5115 ts_tablet_manager.cc:1875] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.931037 5106 tablet_bootstrap.cc:492] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.931207 5106 ts_tablet_manager.cc:1367] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.009s
I20240503 02:12:47.932077 5106 raft_consensus.cc:348] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.932339 5106 raft_consensus.cc:374] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.932410 5106 raft_consensus.cc:729] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.932631 5106 consensus_queue.cc:260] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.934091 5105 tablet_bootstrap.cc:654] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.936332 5106 ts_tablet_manager.cc:1398] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:47.936677 5106 tablet_bootstrap.cc:492] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.938477 5106 tablet_bootstrap.cc:654] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.943493 6830 raft_consensus.cc:482] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.943537 5159 ts_tablet_manager.cc:1888] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:47.943667 5159 log.cc:1196] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fcfb174825f1441f861ae6816137f855
I20240503 02:12:47.943658 6830 raft_consensus.cc:504] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.943986 5159 ts_tablet_manager.cc:1909] T fcfb174825f1441f861ae6816137f855 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:47.944382 6830 leader_election.cc:288] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:47.945238 4608 leader_election.cc:341] T aa67857b858c4719b498291c52b7aceb P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: aa67857b858c4719b498291c52b7aceb
I20240503 02:12:47.947737 5115 ts_tablet_manager.cc:1888] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:47.947875 5115 log.cc:1196] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/3f9994ca5aea48d3bc25c097dddba3c3
I20240503 02:12:47.948242 5115 ts_tablet_manager.cc:1909] T 3f9994ca5aea48d3bc25c097dddba3c3 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:47.954421 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 940) took 3910 ms. Trace:
I20240503 02:12:47.954562 5115 rpcz_store.cc:276] 0503 02:12:44.044376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.044411 (+ 35us) service_pool.cc:225] Handling call
0503 02:12:47.926169 (+3881758us) ts_tablet_manager.cc:1120] Deleting tablet 3f9994ca5aea48d3bc25c097dddba3c3
0503 02:12:47.926180 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.930402 (+ 4222us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.947610 (+ 17208us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:47.954365 (+ 6755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3881634}
I20240503 02:12:47.955153 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 3f9994ca5aea48d3bc25c097dddba3c3 (table default.Table_150a5f1631d94beaa4e6b81e2e9ade3d [id=76c02a0a442c48c886eed51e2e3f58ad]) successfully deleted
I20240503 02:12:47.955552 5115 tablet_replica.cc:330] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:47.955826 5115 raft_consensus.cc:2229] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.956017 5115 raft_consensus.cc:2258] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.956027 6941 raft_consensus.cc:482] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.956979 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fcfb174825f1441f861ae6816137f855 (table default.Table_d377db521b5346bbb8997345606a36e2 [id=b657cc5032354ac688820deb2c891d1a]) successfully deleted
I20240503 02:12:47.956636 5115 ts_tablet_manager.cc:1875] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.957633 5159 tablet_replica.cc:330] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:47.959758 5159 raft_consensus.cc:2229] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:47.959833 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 442d795ee01c417dbdeda41fd6853dd7 (table default.Table_512dd0dd609548978d52d05cc8e69782 [id=52915748a54142109318ab8a12dea14f]). Expected version 1 got 0
I20240503 02:12:47.960057 5159 raft_consensus.cc:2258] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:47.960731 5159 ts_tablet_manager.cc:1875] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:47.956189 6941 raft_consensus.cc:504] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.961949 6941 leader_election.cc:288] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.962476 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73ec6543b9a468492f3b97772ce43f9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.962700 4672 raft_consensus.cc:2454] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 0.
I20240503 02:12:47.963179 4746 leader_election.cc:302] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.963455 6941 raft_consensus.cc:2781] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:47.963559 6941 raft_consensus.cc:482] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:47.963593 5106 tablet_bootstrap.cc:492] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:47.963624 6941 raft_consensus.cc:3037] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.963719 5106 ts_tablet_manager.cc:1367] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20240503 02:12:47.964509 5106 raft_consensus.cc:348] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.964707 5106 raft_consensus.cc:374] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.964763 5106 raft_consensus.cc:729] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.964993 5106 consensus_queue.cc:260] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.966357 6941 raft_consensus.cc:504] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.966590 5106 ts_tablet_manager.cc:1398] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:47.967558 6941 leader_election.cc:288] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:47.969815 6830 consensus_queue.cc:1035] T c099a306a56643069f63e34a5f0d3dd7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.970021 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73ec6543b9a468492f3b97772ce43f9" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81"
I20240503 02:12:47.970209 4672 raft_consensus.cc:3037] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:47.975000 4672 raft_consensus.cc:2454] T a73ec6543b9a468492f3b97772ce43f9 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc6d33969e74c209310005ae911c405 in term 1.
I20240503 02:12:47.975813 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 6a11c306232c4e49ab02be0f195df935 (table default.Table_aa8c3d142c5746cc83cb777a33f38060 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]). Expected version 1 got 0
I20240503 02:12:47.975530 5106 tablet_bootstrap.cc:492] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:47.979014 6941 raft_consensus.cc:482] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:47.979087 4746 leader_election.cc:302] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:47.979367 7003 raft_consensus.cc:2781] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:47.979154 6941 raft_consensus.cc:504] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.979512 7003 raft_consensus.cc:686] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [term 1 LEADER]: Becoming Leader. State: Replica: acc6d33969e74c209310005ae911c405, State: Running, Role: LEADER
I20240503 02:12:47.979853 7003 consensus_queue.cc:237] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.980096 6941 leader_election.cc:288] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:47.979156 5105 tablet_bootstrap.cc:492] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:47.981549 5105 ts_tablet_manager.cc:1367] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.002s
I20240503 02:12:47.982393 5105 raft_consensus.cc:348] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.982586 5105 raft_consensus.cc:374] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.982635 5105 raft_consensus.cc:729] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.982851 5105 consensus_queue.cc:260] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:47.983390 5105 ts_tablet_manager.cc:1398] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:47.983682 5105 tablet_bootstrap.cc:492] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:47.984750 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b6a18be64049049e1a083bf4af51ba" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:47.986187 5105 tablet_bootstrap.cc:654] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.988284 5106 tablet_bootstrap.cc:654] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:47.989459 4746 leader_election.cc:341] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:47.993073 4271 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 6a11c306232c4e49ab02be0f195df935 (table default.Table_aa8c3d142c5746cc83cb777a33f38060 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]). Expected version 1 got 0
I20240503 02:12:48.006168 5115 ts_tablet_manager.cc:1888] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:48.006330 5115 log.cc:1196] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/8d0494ee4bd746069bfaf39bfb46687d
I20240503 02:12:48.006757 5115 ts_tablet_manager.cc:1909] T 8d0494ee4bd746069bfaf39bfb46687d P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.007311 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 941) took 3958 ms. Trace:
I20240503 02:12:48.007455 5115 rpcz_store.cc:276] 0503 02:12:44.048628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.048789 (+ 161us) service_pool.cc:225] Handling call
0503 02:12:47.955497 (+3906708us) ts_tablet_manager.cc:1120] Deleting tablet 8d0494ee4bd746069bfaf39bfb46687d
0503 02:12:47.955507 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:47.960626 (+ 5119us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.006044 (+ 45418us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.007238 (+ 1194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3906428}
I20240503 02:12:48.007897 5115 tablet_replica.cc:330] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.008131 5115 raft_consensus.cc:2229] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.008118 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 8d0494ee4bd746069bfaf39bfb46687d (table default.Table_af6ff741f769452798eade9274479e3d [id=f8d268fc75d444959f9dc2b0989cc235]) successfully deleted
I20240503 02:12:48.008358 5115 raft_consensus.cc:2258] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.008592 6814 raft_consensus.cc:482] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.008757 6814 raft_consensus.cc:504] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.009860 6814 leader_election.cc:288] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:48.010936 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8287b45fa0b492ca82c22164f3a7837" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.011723 5159 ts_tablet_manager.cc:1888] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.011860 5159 log.cc:1196] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a48d5eac44264899a7a372b9f85fb8f5
I20240503 02:12:48.012212 5159 ts_tablet_manager.cc:1909] T a48d5eac44264899a7a372b9f85fb8f5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
W20240503 02:12:48.013178 4746 leader_election.cc:341] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.015029 5115 ts_tablet_manager.cc:1875] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.017488 5106 tablet_bootstrap.cc:492] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.017639 5106 ts_tablet_manager.cc:1367] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.005s
I20240503 02:12:48.017725 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a48d5eac44264899a7a372b9f85fb8f5 (table default.Table_799bd817690849dc9654a7fb51482f2b [id=abda9019ae0142b78b5e5229f5149509]) successfully deleted
I20240503 02:12:48.018333 5106 raft_consensus.cc:348] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.018563 5106 raft_consensus.cc:374] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.018630 5106 raft_consensus.cc:729] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.018841 5106 consensus_queue.cc:260] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.019429 5106 ts_tablet_manager.cc:1398] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:48.020511 4271 catalog_manager.cc:5484] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc6d33969e74c209310005ae911c405 (127.4.17.67). New cstate: current_term: 1 leader_uuid: "acc6d33969e74c209310005ae911c405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:48.021010 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet a73ec6543b9a468492f3b97772ce43f9 (table default.Table_2aec4967b23a414596b8b354cf8de2d9 [id=eb724fd2c11544ab9d8d539d30a2ffa4]). Expected version 2 got 0
I20240503 02:12:48.024638 5106 tablet_bootstrap.cc:492] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:48.024580 4165 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4423: Timed out: CheckLeaks RPC to 127.4.17.65:39837 timed out after 1.000s (SENT)
I20240503 02:12:48.024780 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4423
I20240503 02:12:48.028533 5106 tablet_bootstrap.cc:654] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.029635 5115 ts_tablet_manager.cc:1888] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:48.029753 5115 log.cc:1196] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/2f862891b43a4845b723bfdb9a7da591
I20240503 02:12:48.030094 5115 ts_tablet_manager.cc:1909] T 2f862891b43a4845b723bfdb9a7da591 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.037317 5106 tablet_bootstrap.cc:492] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.037488 5106 ts_tablet_manager.cc:1367] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240503 02:12:48.038158 5106 raft_consensus.cc:348] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.038342 5106 raft_consensus.cc:374] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.038370 5105 tablet_bootstrap.cc:492] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.038398 5106 raft_consensus.cc:729] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.038528 5105 ts_tablet_manager.cc:1367] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.001s
I20240503 02:12:48.038646 5106 consensus_queue.cc:260] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.039288 5106 ts_tablet_manager.cc:1398] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:48.039261 5105 raft_consensus.cc:348] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.039474 5105 raft_consensus.cc:374] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.039547 5105 raft_consensus.cc:729] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.039821 5105 consensus_queue.cc:260] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.040385 5105 ts_tablet_manager.cc:1398] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:48.040697 5105 tablet_bootstrap.cc:492] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:48.043265 4608 consensus_peers.cc:485] T 043ba14eaa5b46319b0b88457d2a5f45 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 043ba14eaa5b46319b0b88457d2a5f45. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.050482 5105 tablet_bootstrap.cc:654] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.050961 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet aa699a1dc3c34be59925560d5e11f49e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:24 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.051074 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce4b33f90c40c48cf0a538a1a7fde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.052520 6899 raft_consensus.cc:482] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.052683 6899 raft_consensus.cc:504] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.052038 5159 tablet_replica.cc:330] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.053225 5159 raft_consensus.cc:2229] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.053386 6899 leader_election.cc:288] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.053436 5159 raft_consensus.cc:2258] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.055246 5159 ts_tablet_manager.cc:1875] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:48.057458 4608 leader_election.cc:341] T d607da9689c14e51add9178a48b35913 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: d607da9689c14e51add9178a48b35913
W20240503 02:12:48.061427 4608 consensus_peers.cc:485] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79aae500e9a842ed90357f4a65fd37ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.065250 4809 raft_consensus.cc:1261] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Refusing update from remote peer 1f50719d94564844b15e1a9211d48a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:48.066031 6899 consensus_queue.cc:1035] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.066308 5106 tablet_bootstrap.cc:492] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.066043 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 942) took 4016 ms. Trace:
I20240503 02:12:48.066632 5115 rpcz_store.cc:276] 0503 02:12:44.049802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.050197 (+ 395us) service_pool.cc:225] Handling call
0503 02:12:48.007789 (+3957592us) ts_tablet_manager.cc:1120] Deleting tablet 2f862891b43a4845b723bfdb9a7da591
0503 02:12:48.007800 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.021509 (+ 13709us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.029513 (+ 8004us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.066017 (+ 36504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3949400}
I20240503 02:12:48.066944 5115 tablet_replica.cc:330] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.067169 5115 raft_consensus.cc:2229] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:48.067453 5115 raft_consensus.cc:2258] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.070331 5106 tablet_bootstrap.cc:654] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.072872 5115 ts_tablet_manager.cc:1875] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.079633 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 2f862891b43a4845b723bfdb9a7da591 (table default.Table_34fcaf75826241bea9fc24cb331f823e [id=4b02cf4906d54d59918e8a5bb8b1e2f6]) successfully deleted
I20240503 02:12:48.084188 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 905ca3be01e344d38350a58ac9f0d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.086899 5159 ts_tablet_manager.cc:1888] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.087069 5159 log.cc:1196] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1ce9d746acfe4e9aaea600293470d373
I20240503 02:12:48.088243 7003 raft_consensus.cc:482] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.088438 7003 raft_consensus.cc:504] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.089184 7003 leader_election.cc:288] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.089627 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3710b653aa24b7ca389aa95d7cdd2fe" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:48.090484 4746 leader_election.cc:341] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.092275 5159 ts_tablet_manager.cc:1909] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.093390 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1ce9d746acfe4e9aaea600293470d373 (table default.Table_033e1f689d894383a2e2c0ebbbc6023b [id=009c481538b643a58c9d625b8a184bd2]) successfully deleted
I20240503 02:12:48.093911 5159 tablet_replica.cc:330] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.096199 5115 ts_tablet_manager.cc:1888] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:48.096343 5115 log.cc:1196] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/73965590a0ea4bffae6e99fa1367fad0
I20240503 02:12:48.096720 5115 ts_tablet_manager.cc:1909] T 73965590a0ea4bffae6e99fa1367fad0 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.096738 5159 raft_consensus.cc:2229] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.097172 5159 raft_consensus.cc:2258] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.097662 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 944) took 3935 ms. Trace:
I20240503 02:12:48.097770 5115 rpcz_store.cc:276] 0503 02:12:44.162604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.162654 (+ 50us) service_pool.cc:225] Handling call
0503 02:12:48.066898 (+3904244us) ts_tablet_manager.cc:1120] Deleting tablet 73965590a0ea4bffae6e99fa1367fad0
0503 02:12:48.066907 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.076450 (+ 9543us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.096050 (+ 19600us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.097633 (+ 1583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3904009}
I20240503 02:12:48.098191 5159 ts_tablet_manager.cc:1875] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.098331 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 73965590a0ea4bffae6e99fa1367fad0 (table default.Table_9709e189b5424f23a7663146d1a912ac [id=5dfeefb7aef7484d8c5c9df2189f8932]) successfully deleted
I20240503 02:12:48.105409 6899 raft_consensus.cc:482] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.105620 6899 raft_consensus.cc:504] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.106325 6899 leader_election.cc:288] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.106710 5106 tablet_bootstrap.cc:492] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.106850 5106 ts_tablet_manager.cc:1367] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.010s
I20240503 02:12:48.107515 5106 raft_consensus.cc:348] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.107708 5106 raft_consensus.cc:374] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.107766 5106 raft_consensus.cc:729] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.108037 5106 consensus_queue.cc:260] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.108606 5106 ts_tablet_manager.cc:1398] T a48d5eac44264899a7a372b9f85fb8f5 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:48.111135 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efef931ca554ae18579bbea8d1c04f7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.111461 4809 raft_consensus.cc:2454] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:48.111980 4608 leader_election.cc:302] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.112324 6899 raft_consensus.cc:2781] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.112416 6899 raft_consensus.cc:482] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.112506 6899 raft_consensus.cc:3037] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.113009 6814 raft_consensus.cc:482] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.113201 6814 raft_consensus.cc:504] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.113910 6814 leader_election.cc:288] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.115005 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86ac17982324847949b8a9245329489" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
W20240503 02:12:48.116216 4746 leader_election.cc:341] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.117377 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc]). Expected version 2 got 0
I20240503 02:12:48.118542 5106 tablet_bootstrap.cc:492] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.120149 5106 tablet_bootstrap.cc:654] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.121639 6899 raft_consensus.cc:504] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.122295 6899 leader_election.cc:288] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.122522 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 945) took 3923 ms. Trace:
I20240503 02:12:48.122664 5115 rpcz_store.cc:276] 0503 02:12:44.198680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.198737 (+ 57us) service_pool.cc:225] Handling call
0503 02:12:48.122291 (+3923554us) ts_tablet_manager.cc:1120] Deleting tablet b9470480a9de45209044f8e8c461519a
0503 02:12:48.122302 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.122498 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3923286}
I20240503 02:12:48.123145 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 946) took 3923 ms. Trace:
I20240503 02:12:48.123131 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efef931ca554ae18579bbea8d1c04f7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:48.123368 4805 raft_consensus.cc:3037] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.126082 4805 raft_consensus.cc:2454] T 7efef931ca554ae18579bbea8d1c04f7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:48.130102 7016 tablet.cc:1722] T cf10c48d165047668fbd5ca17f917fe3 P acc6d33969e74c209310005ae911c405: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:48.130635 5115 rpcz_store.cc:276] 0503 02:12:44.199928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.199990 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:48.123036 (+3923046us) ts_tablet_manager.cc:1120] Deleting tablet 8d1ac81e1a6f41cf974bb30e6b74eea2
0503 02:12:48.123045 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.123129 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3922836}
I20240503 02:12:48.131230 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 943) took 3981 ms. Trace:
I20240503 02:12:48.126639 4608 leader_election.cc:302] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:48.132495 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1ac81e1a6f41cf974bb30e6b74eea2 already in progress: creating tablet
I20240503 02:12:48.132983 6897 raft_consensus.cc:2781] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.133112 6897 raft_consensus.cc:686] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:48.133355 6897 consensus_queue.cc:237] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.137856 5106 tablet_bootstrap.cc:492] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.138012 5106 ts_tablet_manager.cc:1367] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20240503 02:12:48.138696 5106 raft_consensus.cc:348] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.138932 5106 raft_consensus.cc:374] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.138998 5106 raft_consensus.cc:729] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.139303 5106 consensus_queue.cc:260] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.149677 5106 ts_tablet_manager.cc:1398] T b8474c841cb64373a223ad76b43ecf5f P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.012s user 0.001s sys 0.006s
I20240503 02:12:48.150015 5106 tablet_bootstrap.cc:492] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.151693 5106 tablet_bootstrap.cc:654] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.152101 7015 tablet.cc:1722] T cf10c48d165047668fbd5ca17f917fe3 P 1f50719d94564844b15e1a9211d48a81: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 02:12:48.152740 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet cf10c48d165047668fbd5ca17f917fe3 (table default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc]). Expected version 2 got 0
I20240503 02:12:48.153635 5105 tablet_bootstrap.cc:492] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.153793 5105 ts_tablet_manager.cc:1367] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.019s
I20240503 02:12:48.154105 7003 raft_consensus.cc:482] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.154269 7003 raft_consensus.cc:504] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.154707 5105 raft_consensus.cc:348] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.154919 5105 raft_consensus.cc:374] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.154955 7003 leader_election.cc:288] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.155009 5105 raft_consensus.cc:729] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.155305 5105 consensus_queue.cc:260] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.155880 5105 ts_tablet_manager.cc:1398] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:48.156626 5105 tablet_bootstrap.cc:492] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.158437 5105 tablet_bootstrap.cc:654] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.160578 5106 tablet_bootstrap.cc:492] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.160748 5106 ts_tablet_manager.cc:1367] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240503 02:12:48.161497 5106 raft_consensus.cc:348] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.161739 5106 raft_consensus.cc:374] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.161798 5106 raft_consensus.cc:729] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.162065 5106 consensus_queue.cc:260] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.162631 5106 ts_tablet_manager.cc:1398] T 1ce9d746acfe4e9aaea600293470d373 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 02:12:48.162858 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 452d494e14c440e6805055ba333b4e7f (table default.Table_1f82b7031a824857bfc325fc0c6b1292 [id=3f70407631dd4d3c9a303411aaeaeaba], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 452d494e14c440e6805055ba333b4e7f consensus state has no leader
I20240503 02:12:48.162930 5106 tablet_bootstrap.cc:492] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.163738 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b3d1e7ce83425db0bd56457d9f2544 with delete_type TABLET_DATA_DELETED (replaced by tablet 95c7e4a2954e4161b9acc029873f923b) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.165329 5106 tablet_bootstrap.cc:654] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.167186 5105 tablet_bootstrap.cc:492] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.167363 5105 ts_tablet_manager.cc:1367] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:48.168222 5105 raft_consensus.cc:348] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.168484 5105 raft_consensus.cc:374] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.168581 5105 raft_consensus.cc:729] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.168826 5105 consensus_queue.cc:260] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.169637 5105 ts_tablet_manager.cc:1398] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:12:48.169800 5115 rpcz_store.cc:276] 0503 02:12:44.150160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.150226 (+ 66us) service_pool.cc:225] Handling call
0503 02:12:48.131084 (+3980858us) ts_tablet_manager.cc:1120] Deleting tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2
0503 02:12:48.131094 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.131210 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3929444}
I20240503 02:12:48.170317 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 947) took 3964 ms. Trace:
I20240503 02:12:48.170423 5115 rpcz_store.cc:276] 0503 02:12:44.205485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.205771 (+ 286us) service_pool.cc:225] Handling call
0503 02:12:48.170163 (+3964392us) ts_tablet_manager.cc:1120] Deleting tablet 9d900a0dc6c34076a0993210693a57a0
0503 02:12:48.170176 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.170296 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3964145}
W20240503 02:12:48.170513 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa94ddd7d7d4fdb945d444e82fdc8b2 already in progress: creating tablet
I20240503 02:12:48.170754 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 948) took 3962 ms. Trace:
W20240503 02:12:48.170969 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9d900a0dc6c34076a0993210693a57a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d900a0dc6c34076a0993210693a57a0 already in progress: creating tablet
I20240503 02:12:48.170836 5115 rpcz_store.cc:276] 0503 02:12:44.208772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.209396 (+ 624us) service_pool.cc:225] Handling call
0503 02:12:48.170653 (+3961257us) ts_tablet_manager.cc:1120] Deleting tablet 7c19263f06ec41d7a6fe6a8df7f88f8f
0503 02:12:48.170663 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.170738 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3961026}
I20240503 02:12:48.171269 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 949) took 3954 ms. Trace:
W20240503 02:12:48.171386 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7c19263f06ec41d7a6fe6a8df7f88f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c19263f06ec41d7a6fe6a8df7f88f8f already in progress: creating tablet
I20240503 02:12:48.171355 5115 rpcz_store.cc:276] 0503 02:12:44.217617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.217685 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:48.171168 (+3953483us) ts_tablet_manager.cc:1120] Deleting tablet 89f3232d509f4cb39eb8923fe1d120a7
0503 02:12:48.171176 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.171254 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3953120}
I20240503 02:12:48.171669 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 950) took 3784 ms. Trace:
W20240503 02:12:48.171763 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 89f3232d509f4cb39eb8923fe1d120a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f3232d509f4cb39eb8923fe1d120a7 already in progress: creating tablet
I20240503 02:12:48.171746 5115 rpcz_store.cc:276] 0503 02:12:44.387117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.387192 (+ 75us) service_pool.cc:225] Handling call
0503 02:12:48.171582 (+3784390us) ts_tablet_manager.cc:1120] Deleting tablet 1b13ea83ffa54dcbbfebaaa76e1686a9
0503 02:12:48.171589 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.171656 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3784167}
I20240503 02:12:48.172027 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 954) took 3536 ms. Trace:
W20240503 02:12:48.172138 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 already in progress: creating tablet
I20240503 02:12:48.172109 5115 rpcz_store.cc:276] 0503 02:12:44.635458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.636017 (+ 559us) service_pool.cc:225] Handling call
0503 02:12:48.171932 (+3535915us) ts_tablet_manager.cc:1120] Deleting tablet 7219ce14a9ae4488b97914b9d02995f9
0503 02:12:48.171939 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.172013 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3535706}
I20240503 02:12:48.172346 5115 tablet_replica.cc:330] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:48.172523 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7219ce14a9ae4488b97914b9d02995f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7219ce14a9ae4488b97914b9d02995f9 already in progress: creating tablet
I20240503 02:12:48.172569 5115 raft_consensus.cc:2229] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.172748 5115 raft_consensus.cc:2258] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.173321 5115 ts_tablet_manager.cc:1875] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.174733 5159 ts_tablet_manager.cc:1888] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.176285 5159 log.cc:1196] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/aa699a1dc3c34be59925560d5e11f49e
I20240503 02:12:48.176999 6899 raft_consensus.cc:482] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.177146 6899 raft_consensus.cc:504] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.177819 6899 leader_election.cc:288] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.178455 5159 ts_tablet_manager.cc:1909] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.182588 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11c306232c4e49ab02be0f195df935" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.182880 4810 raft_consensus.cc:2454] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:48.183594 4608 leader_election.cc:302] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.184743 6899 raft_consensus.cc:2781] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.184904 6899 raft_consensus.cc:482] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.185365 6899 raft_consensus.cc:3037] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.185281 5106 tablet_bootstrap.cc:492] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.186012 5106 ts_tablet_manager.cc:1367] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.010s
I20240503 02:12:48.189303 5106 raft_consensus.cc:348] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.189481 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet a48d5eac44264899a7a372b9f85fb8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.192114 5106 raft_consensus.cc:374] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.192332 5106 raft_consensus.cc:729] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.192670 5106 consensus_queue.cc:260] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.193441 5106 ts_tablet_manager.cc:1398] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:48.193797 5106 tablet_bootstrap.cc:492] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.194670 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca69a5b48754602b04481fb6a94864f" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.194846 6948 raft_consensus.cc:482] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.194729 6897 raft_consensus.cc:482] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.194943 6948 raft_consensus.cc:504] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.195005 6897 raft_consensus.cc:504] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.195508 6948 leader_election.cc:288] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.195557 6897 leader_election.cc:288] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.196206 5105 tablet_bootstrap.cc:492] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:48.197167 4608 consensus_peers.cc:485] T ec641e403ed843fda31dabd3920aeb68 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec641e403ed843fda31dabd3920aeb68. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.197984 5105 tablet_bootstrap.cc:654] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.198103 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa699a1dc3c34be59925560d5e11f49e" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.198127 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d795ee01c417dbdeda41fd6853dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.198402 4805 raft_consensus.cc:2454] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
W20240503 02:12:48.198830 4608 leader_election.cc:341] T aa699a1dc3c34be59925560d5e11f49e P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240503 02:12:48.199059 4608 leader_election.cc:302] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.199481 6948 raft_consensus.cc:2781] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.199568 6948 raft_consensus.cc:482] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.199674 6948 raft_consensus.cc:3037] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.202380 6948 raft_consensus.cc:504] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.203053 6948 leader_election.cc:288] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.204033 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442d795ee01c417dbdeda41fd6853dd7" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:48.204267 4810 raft_consensus.cc:3037] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.207139 4810 raft_consensus.cc:2454] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:48.207844 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b91a3f3cad74f6aa6035a013cca651f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:30 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.208338 4608 leader_election.cc:302] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.208690 6948 raft_consensus.cc:2781] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.209002 6948 raft_consensus.cc:686] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:48.209579 6948 consensus_queue.cc:237] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.209885 7003 raft_consensus.cc:482] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.210033 7003 raft_consensus.cc:504] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.210675 7003 leader_election.cc:288] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.210767 6899 raft_consensus.cc:504] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.215795 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11c306232c4e49ab02be0f195df935" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:48.216114 4810 raft_consensus.cc:3037] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.217623 5105 tablet_bootstrap.cc:492] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.217815 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1091) took 1085 ms. Trace:
I20240503 02:12:48.217954 5159 rpcz_store.cc:276] 0503 02:12:47.132168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.132245 (+ 77us) service_pool.cc:225] Handling call
0503 02:12:48.093822 (+961577us) ts_tablet_manager.cc:1120] Deleting tablet aa699a1dc3c34be59925560d5e11f49e
0503 02:12:48.093837 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.167853 (+ 74016us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.174560 (+ 6707us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.217762 (+ 43202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961315}
I20240503 02:12:48.218418 5159 tablet_replica.cc:330] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.218585 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet aa699a1dc3c34be59925560d5e11f49e (table default.Table_cee118c15da24103b1ad3d2b2daf3f52 [id=e3260d7fe9364cc0b19c9caf9a55a94f]) successfully deleted
I20240503 02:12:48.218672 5159 raft_consensus.cc:2229] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.218856 5159 raft_consensus.cc:2258] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.219349 7003 raft_consensus.cc:482] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.219506 7003 raft_consensus.cc:504] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.219965 4810 raft_consensus.cc:2454] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:48.220244 7003 leader_election.cc:288] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.220420 4608 leader_election.cc:302] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.220737 7019 raft_consensus.cc:2781] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.217801 5105 ts_tablet_manager.cc:1367] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 02:12:48.220948 7019 raft_consensus.cc:686] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:48.221282 7019 consensus_queue.cc:237] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.221590 5105 raft_consensus.cc:348] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.221803 5105 raft_consensus.cc:374] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.221885 5105 raft_consensus.cc:729] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.222152 5105 consensus_queue.cc:260] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.223016 5105 ts_tablet_manager.cc:1398] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:48.223632 5115 ts_tablet_manager.cc:1888] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.224527 5115 log.cc:1196] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/20ce11c70a924b8d92d8f36fc7401dfd
I20240503 02:12:48.225109 5115 ts_tablet_manager.cc:1909] T 20ce11c70a924b8d92d8f36fc7401dfd P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.226454 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 955) took 3590 ms. Trace:
I20240503 02:12:48.226881 6830 raft_consensus.cc:482] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.228008 6830 raft_consensus.cc:504] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.228854 6830 leader_election.cc:288] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.229077 5115 rpcz_store.cc:276] 0503 02:12:44.635562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.636258 (+ 696us) service_pool.cc:225] Handling call
0503 02:12:48.172301 (+3536043us) ts_tablet_manager.cc:1120] Deleting tablet 20ce11c70a924b8d92d8f36fc7401dfd
0503 02:12:48.172309 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.176449 (+ 4140us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.223507 (+ 47058us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.226430 (+ 2923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3535898}
I20240503 02:12:48.229691 7026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:48.229911 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54959b3e30944fab8bd5fd47ce42d246" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.230103 5105 tablet_bootstrap.cc:492] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.230574 4810 raft_consensus.cc:2454] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:48.232088 5105 tablet_bootstrap.cc:654] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.231017 4608 leader_election.cc:302] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.233017 6830 raft_consensus.cc:2781] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.233109 6830 raft_consensus.cc:482] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.233187 6830 raft_consensus.cc:3037] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.233928 5106 tablet_bootstrap.cc:654] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.235862 6830 raft_consensus.cc:504] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.236543 6830 leader_election.cc:288] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.232215 6899 leader_election.cc:288] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.237354 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 20ce11c70a924b8d92d8f36fc7401dfd (table default.Table_4f1cb580826c474cb8690af6403aefdf [id=a73d84ef3a5a4155afeea6bbc4476a16]) successfully deleted
I20240503 02:12:48.237850 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54959b3e30944fab8bd5fd47ce42d246" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:48.238191 4810 raft_consensus.cc:3037] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.238193 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 956) took 3602 ms. Trace:
I20240503 02:12:48.237732 7026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
W20240503 02:12:48.238706 4746 leader_election.cc:341] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.239801 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038f11066dc34ba9a833f84a54ff20bf" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.239802 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1245 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:48.240331 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c7ba3639bd4cb4b897dc9aa47ff8c3" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.238282 5115 rpcz_store.cc:276] 0503 02:12:44.635647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.636430 (+ 783us) service_pool.cc:225] Handling call
0503 02:12:48.237339 (+3600909us) ts_tablet_manager.cc:1120] Deleting tablet f146f7cce0b24d3087bfe0973e2b7ef0
0503 02:12:48.237351 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.238172 (+ 821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3600802}
W20240503 02:12:48.241060 4259 catalog_manager.cc:4797] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet f146f7cce0b24d3087bfe0973e2b7ef0 because the tablet was not found. No further retry: Not found: Tablet not found: f146f7cce0b24d3087bfe0973e2b7ef0
W20240503 02:12:48.241266 4746 leader_election.cc:341] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.241521 4810 raft_consensus.cc:2454] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:48.241600 5115 tablet_replica.cc:330] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
W20240503 02:12:48.241722 4746 leader_election.cc:341] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.243736 5159 ts_tablet_manager.cc:1875] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.244203 4608 leader_election.cc:302] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.244686 6899 raft_consensus.cc:2781] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.244850 6899 raft_consensus.cc:686] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:48.245056 5115 raft_consensus.cc:2229] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.246063 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1245 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:48.247222 6193 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:48.253155 4271 catalog_manager.cc:6029] T 00000000000000000000000000000000 P df63329dc2b149d9b08cea341b540d1d: default.Table_958b53896e2945d7a5fc23d65f0dc378 [id=156d16ce2a4246cd81968e098a9227dc] alter complete (version 2)
I20240503 02:12:48.255223 7003 raft_consensus.cc:482] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.255389 7003 raft_consensus.cc:504] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.256047 7003 leader_election.cc:288] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.257478 5115 raft_consensus.cc:2258] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.257853 7026 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:48.255128 5106 tablet_bootstrap.cc:492] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.258380 5106 ts_tablet_manager.cc:1367] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.000s
I20240503 02:12:48.259130 5106 raft_consensus.cc:348] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.259343 5106 raft_consensus.cc:374] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.259405 5106 raft_consensus.cc:729] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.259634 5106 consensus_queue.cc:260] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.259724 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e1c2fa9cb3481dafb9d7f1e826a30c" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.260140 5106 ts_tablet_manager.cc:1398] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 02:12:48.260213 4746 leader_election.cc:341] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.260419 5106 tablet_bootstrap.cc:492] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.245146 6899 consensus_queue.cc:237] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.262490 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet ed4e84a639b2403082b737d073f00ad7 (table default.Table_375af786345e47b69fba154241e880c7 [id=3cb9bed8bee34911aab76b39eaa68dae]). Expected version 1 got 0
I20240503 02:12:48.258417 7026 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:12:48.264173 5106 tablet_bootstrap.cc:654] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.259160 5115 ts_tablet_manager.cc:1875] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.271843 5159 ts_tablet_manager.cc:1888] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.272044 5159 log.cc:1196] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b8474c841cb64373a223ad76b43ecf5f
I20240503 02:12:48.272538 5159 ts_tablet_manager.cc:1909] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.273622 4268 catalog_manager.cc:5484] T 442d795ee01c417dbdeda41fd6853dd7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:48.274305 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 442d795ee01c417dbdeda41fd6853dd7 (table default.Table_512dd0dd609548978d52d05cc8e69782 [id=52915748a54142109318ab8a12dea14f]). Expected version 1 got 0
I20240503 02:12:48.274693 4268 catalog_manager.cc:5484] T 7efef931ca554ae18579bbea8d1c04f7 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:48.275023 5105 tablet_bootstrap.cc:492] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.275200 5105 ts_tablet_manager.cc:1367] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240503 02:12:48.275233 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 7efef931ca554ae18579bbea8d1c04f7 (table default.Table_9091ef41bcba4103811266c2b3782904 [id=fde711a2c8e04377996a7cd87f629e68]). Expected version 2 got 0
I20240503 02:12:48.275583 4268 catalog_manager.cc:5484] T 6a11c306232c4e49ab02be0f195df935 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:48.275983 5105 raft_consensus.cc:348] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.276115 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 6a11c306232c4e49ab02be0f195df935 (table default.Table_aa8c3d142c5746cc83cb777a33f38060 [id=ab17dcc9e61b48a09b3c4cbc9b07964b]). Expected version 1 got 0
I20240503 02:12:48.276211 5105 raft_consensus.cc:374] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.276278 5105 raft_consensus.cc:729] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.276504 5105 consensus_queue.cc:260] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.276672 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 1f3dbcbc572f4215b7f59f3750a4b9fa (table default.Table_910d49929ead4e98b7c7a3d1340a6ca1 [id=45383bf6b0004d8d8f059f46fbc0b4ee]). Expected version 3 got 0
I20240503 02:12:48.284060 7003 consensus_queue.cc:1035] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.297281 5115 ts_tablet_manager.cc:1888] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:48.297559 5115 log.cc:1196] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/c1f39e8d17b44895bd5c979d82145ad9
I20240503 02:12:48.298007 5115 ts_tablet_manager.cc:1909] T c1f39e8d17b44895bd5c979d82145ad9 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.298355 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 818788a24cd1419980c0d622fb05683b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.298892 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 957) took 3663 ms. Trace:
I20240503 02:12:48.299067 5115 rpcz_store.cc:276] 0503 02:12:44.635731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.636566 (+ 835us) service_pool.cc:225] Handling call
0503 02:12:48.241546 (+3604980us) ts_tablet_manager.cc:1120] Deleting tablet c1f39e8d17b44895bd5c979d82145ad9
0503 02:12:48.241557 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.293107 (+ 51550us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.297163 (+ 4056us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.298861 (+ 1698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3604840}
I20240503 02:12:48.299669 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet c1f39e8d17b44895bd5c979d82145ad9 (table default.Table_9b099c5cc2d14c2ba0fad6df023e947c [id=ec6cee0c5c6c466682653bba626dc0c7]) successfully deleted
I20240503 02:12:48.301756 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 967) took 3646 ms. Trace:
I20240503 02:12:48.301854 5115 rpcz_store.cc:276] 0503 02:12:44.655344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.656293 (+ 949us) service_pool.cc:225] Handling call
0503 02:12:48.300002 (+3643709us) ts_tablet_manager.cc:1120] Deleting tablet 9003cc376aa149b59947bc32bd071163
0503 02:12:48.301613 (+ 1611us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.301735 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3643536}
I20240503 02:12:48.302139 5115 tablet_replica.cc:330] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.302325 5115 raft_consensus.cc:2229] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.302616 5115 raft_consensus.cc:2258] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.303272 5115 ts_tablet_manager.cc:1875] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.303350 5105 ts_tablet_manager.cc:1398] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20240503 02:12:48.303755 5105 tablet_bootstrap.cc:492] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:48.304741 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 9003cc376aa149b59947bc32bd071163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9003cc376aa149b59947bc32bd071163 already in progress: creating tablet
I20240503 02:12:48.313176 5105 tablet_bootstrap.cc:654] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.314397 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b8474c841cb64373a223ad76b43ecf5f (table default.Table_e6ff638542714113aa75b6b69e0de916 [id=c8726bc562f84a739911a48a1b9f1765]) successfully deleted
I20240503 02:12:48.317759 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet b8474c841cb64373a223ad76b43ecf5f with delete_type TABLET_DATA_DELETED (replaced by tablet 73198ec769a9486da3f85e8fcbef8e78) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.320026 5159 tablet_replica.cc:330] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.320303 5159 raft_consensus.cc:2229] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.320508 5159 raft_consensus.cc:2258] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.321096 5159 ts_tablet_manager.cc:1875] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.325075 5106 tablet_bootstrap.cc:492] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.325376 5106 ts_tablet_manager.cc:1367] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.006s
I20240503 02:12:48.326056 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet b8287b45fa0b492ca82c22164f3a7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.326651 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce9d746acfe4e9aaea600293470d373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.326539 5106 raft_consensus.cc:348] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.327069 5106 raft_consensus.cc:374] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.327180 5106 raft_consensus.cc:729] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.327556 5106 consensus_queue.cc:260] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.333112 4268 catalog_manager.cc:5484] T 54959b3e30944fab8bd5fd47ce42d246 P 1f50719d94564844b15e1a9211d48a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f50719d94564844b15e1a9211d48a81 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "1f50719d94564844b15e1a9211d48a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:48.333653 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 54959b3e30944fab8bd5fd47ce42d246 (table default.Table_db80a3085448432ca09aafe2ba19dc90 [id=aaefcb18de5b40f1b46a3ffe9bed832a]). Expected version 2 got 0
I20240503 02:12:48.338487 5106 ts_tablet_manager.cc:1398] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:48.338922 5106 tablet_bootstrap.cc:492] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.340721 5115 ts_tablet_manager.cc:1888] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240503 02:12:48.342576 5115 log.cc:1196] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/f40706400f114def9e14e6d1d7114962
I20240503 02:12:48.343758 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd296bb3b574d6cbb0252ad10ada00c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:35 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.348486 5115 ts_tablet_manager.cc:1909] T f40706400f114def9e14e6d1d7114962 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.351092 5105 tablet_bootstrap.cc:492] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.355181 5105 ts_tablet_manager.cc:1367] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.007s
I20240503 02:12:48.350211 5159 ts_tablet_manager.cc:1888] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.355464 5159 log.cc:1196] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5cd296bb3b574d6cbb0252ad10ada00c
I20240503 02:12:48.355893 5159 ts_tablet_manager.cc:1909] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.356181 5105 raft_consensus.cc:348] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.356405 5105 raft_consensus.cc:374] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.356496 5105 raft_consensus.cc:729] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.356812 5105 consensus_queue.cc:260] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.357442 5105 ts_tablet_manager.cc:1398] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:48.357940 5105 tablet_bootstrap.cc:492] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.359364 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 968) took 3703 ms. Trace:
I20240503 02:12:48.359905 5105 tablet_bootstrap.cc:654] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.363857 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 676d5f52102d496081df3964722b286f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.355268 6899 raft_consensus.cc:482] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.349859 5106 tablet_bootstrap.cc:654] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.366371 6899 raft_consensus.cc:504] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.367117 6899 leader_election.cc:288] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.368032 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8287b45fa0b492ca82c22164f3a7837" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405" is_pre_election: true
I20240503 02:12:48.368335 4810 raft_consensus.cc:2454] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 0.
I20240503 02:12:48.368461 5115 rpcz_store.cc:276] 0503 02:12:44.655430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.656500 (+ 1070us) service_pool.cc:225] Handling call
0503 02:12:48.302098 (+3645598us) ts_tablet_manager.cc:1120] Deleting tablet f40706400f114def9e14e6d1d7114962
0503 02:12:48.302107 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.331095 (+ 28988us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.340587 (+ 9492us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.359335 (+ 18748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3645480}
I20240503 02:12:48.368917 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 969) took 3713 ms. Trace:
I20240503 02:12:48.369026 5115 rpcz_store.cc:276] 0503 02:12:44.655510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.659246 (+ 3736us) service_pool.cc:225] Handling call
0503 02:12:48.368787 (+3709541us) ts_tablet_manager.cc:1120] Deleting tablet 7f88d274b7654ba4942e757ba3302101
0503 02:12:48.368798 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.368901 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3709321}
I20240503 02:12:48.369319 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 970) took 3709 ms. Trace:
I20240503 02:12:48.369397 5115 rpcz_store.cc:276] 0503 02:12:44.659746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:44.660437 (+ 691us) service_pool.cc:225] Handling call
0503 02:12:48.369218 (+3708781us) ts_tablet_manager.cc:1120] Deleting tablet bd4940fd3c16424eaaaef59ab50a04a8
0503 02:12:48.369225 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.369306 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3708591}
I20240503 02:12:48.369611 5115 tablet_replica.cc:330] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.369566 4608 leader_election.cc:302] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
I20240503 02:12:48.369796 5115 raft_consensus.cc:2229] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:48.370079 5115 raft_consensus.cc:2258] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.370093 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet f40706400f114def9e14e6d1d7114962 (table default.Table_eae0f8da0ab0497e8731eb30c28c8c7c [id=76d119b7d82c4aaba94801c863535173]) successfully deleted
W20240503 02:12:48.370391 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet 7f88d274b7654ba4942e757ba3302101 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f88d274b7654ba4942e757ba3302101 already in progress: creating tablet
W20240503 02:12:48.370599 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet bd4940fd3c16424eaaaef59ab50a04a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4940fd3c16424eaaaef59ab50a04a8 already in progress: creating tablet
I20240503 02:12:48.370658 5115 ts_tablet_manager.cc:1875] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.371495 6899 raft_consensus.cc:2781] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.371580 6899 raft_consensus.cc:482] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.371634 6899 raft_consensus.cc:3037] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.375326 6899 raft_consensus.cc:504] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.376716 6948 consensus_queue.cc:1035] T 83a1d5b2de8d4d3ba5190750545c9f7b P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.379168 6899 leader_election.cc:288] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Requested vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:47.489084 6238 rpcz_store.cc:276] 0503 02:12:45.737325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.737540 (+ 215us) service_pool.cc:225] Handling call
0503 02:12:45.771061 (+ 33521us) raft_consensus.cc:1457] Updating replica for 364 ops
0503 02:12:45.791481 (+ 20420us) raft_consensus.cc:1520] Early marking committed up to index 881
0503 02:12:45.823909 (+ 32428us) raft_consensus.cc:1525] Triggering prepare for 364 ops
0503 02:12:45.824070 (+ 161us) write_op.cc:270] Start()
0503 02:12:45.824111 (+ 41us) write_op.cc:276] Timestamp: P: 1714702357805645 usec, L: 0
0503 02:12:45.913433 (+ 89322us) write_op.cc:270] Start()
0503 02:12:45.913481 (+ 48us) write_op.cc:276] Timestamp: P: 1714702357816224 usec, L: 0
0503 02:12:45.913706 (+ 225us) write_op.cc:270] Start()
0503 02:12:45.913729 (+ 23us) write_op.cc:276] Timestamp: P: 1714702357827923 usec, L: 0
0503 02:12:45.913920 (+ 191us) write_op.cc:270] Start()
0503 02:12:45.913943 (+ 23us) write_op.cc:276] Timestamp: P: 1714702357838621 usec, L: 0
0503 02:12:45.914077 (+ 134us) write_op.cc:270] Start()
0503 02:12:45.914098 (+ 21us) write_op.cc:276] Timestamp: P: 1714702357863868 usec, L: 0
0503 02:12:45.914237 (+ 139us) write_op.cc:270] Start()
0503 02:12:45.914259 (+ 22us) write_op.cc:276] Timestamp: P: 1714702357897886 usec, L: 0
0503 02:12:45.914429 (+ 170us) write_op.cc:270] Start()
0503 02:12:45.914470 (+ 41us) write_op.cc:276] Timestamp: P: 1714702357908182 usec, L: 0
0503 02:12:45.914630 (+ 160us) write_op.cc:270] Start()
0503 02:12:45.914649 (+ 19us) write_op.cc:276] Timestamp: P: 1714702357917534 usec, L: 0
0503 02:12:45.914790 (+ 141us) write_op.cc:270] Start()
0503 02:12:45.914807 (+ 17us) write_op.cc:276] Timestamp: P: 1714702357919952 usec, L: 0
0503 02:12:45.914923 (+ 116us) write_op.cc:270] Start()
0503 02:12:45.914942 (+ 19us) write_op.cc:276] Timestamp: P: 1714702357920821 usec, L: 0
0503 02:12:45.915047 (+ 105us) write_op.cc:270] Start()
0503 02:12:45.915065 (+ 18us) write_op.cc:276] Timestamp: P: 1714702357924006 usec, L: 0
0503 02:12:45.915181 (+ 116us) write_op.cc:270] Start()
0503 02:12:45.915202 (+ 21us) write_op.cc:276] Timestamp: P: 1714702357952107 usec, L: 0
0503 02:12:45.915340 (+ 138us) write_op.cc:270] Start()
0503 02:12:45.915361 (+ 21us) write_op.cc:276] Timestamp: P: 1714702357952860 usec, L: 0
0503 02:12:45.915521 (+ 160us) write_op.cc:270] Start()
0503 02:12:45.915546 (+ 25us) write_op.cc:276] Timestamp: P: 1714702357953747 usec, L: 0
0503 02:12:45.915675 (+ 129us) write_op.cc:270] Start()
0503 02:12:45.915694 (+ 19us) write_op.cc:276] Timestamp: P: 1714702357954327 usec, L: 0
0503 02:12:45.915808 (+ 114us) write_op.cc:270] Start()
0503 02:12:45.915828 (+ 20us) write_op.cc:276] Timestamp: P: 1714702357954852 usec, L: 0
0503 02:12:45.915967 (+ 139us) write_op.cc:270] Start()
0503 02:12:45.915987 (+ 20us) write_op.cc:276] Timestamp: P: 1714702357969881 usec, L: 0
0503 02:12:45.916127 (+ 140us) write_op.cc:270] Start()
0503 02:12:45.916157 (+ 30us) write_op.cc:276] Timestamp: P: 1714702357979259 usec, L: 0
0503 02:12:45.916268 (+ 111us) write_op.cc:270] Start()
0503 02:12:45.916286 (+ 18us) write_op.cc:276] Timestamp: P: 1714702358072787 usec, L: 0
0503 02:12:45.916421 (+ 135us) write_op.cc:270] Start()
0503 02:12:45.916442 (+ 21us) write_op.cc:276] Timestamp: P: 1714702358290699 usec, L: 0
0503 02:12:45.916574 (+ 132us) write_op.cc:270] Start()
0503 02:12:45.916592 (+ 18us) write_op.cc:276] Timestamp: P: 1714702358332152 usec, L: 0
0503 02:12:45.916727 (+ 135us) write_op.cc:270] Start()
0503 02:12:45.916746 (+ 19us) write_op.cc:276] Timestamp: P: 1714702358338857 usec, L: 0
0503 02:12:45.916871 (+ 125us) write_op.cc:270] Start()
0503 02:12:45.916889 (+ 18us) write_op.cc:276] Timestamp: P: 1714702358340802 usec, L: 0
0503 02:12:45.941176 (+ 24287us) write_op.cc:270] Start()
0503 02:12:45.941212 (+ 36us) write_op.cc:276] Timestamp: P: 1714702358341398 usec, L: 0
0503 02:12:45.941460 (+ 248us) write_op.cc:270] Start()
0503 02:12:45.941484 (+ 24us) write_op.cc:276] Timestamp: P: 1714702358342667 usec, L: 0
0503 02:12:45.941614 (+ 130us) write_op.cc:270] Start()
0503 02:12:45.941638 (+ 24us) write_op.cc:276] Timestamp: P: 1714702358355306 usec, L: 0
0503 02:12:45.941761 (+ 123us) write_op.cc:270] Start()
0503 02:12:45.941777 (+ 16us) write_op.cc:276] Timestamp: P: 1714702358379246 usec, L: 0
0503 02:12:45.941888 (+ 111us) write_op.cc:270] Start()
0503 02:12:45.941902 (+ 14us) write_op.cc:276] Timestamp: P: 1714702358392829 usec, L: 0
0503 02:12:45.942001 (+ 99us) write_op.cc:270] Start()
0503 02:12:45.942015 (+ 14us) write_op.cc:276] Timestamp: P: 1714702358406325 usec, L: 0
0503 02:12:45.942119 (+ 104us) write_op.cc:270] Start()
0503 02:12:45.942134 (+ 15us) write_op.cc:276] Timestamp: P: 1714702358416124 usec, L: 0
0503 02:12:45.942251 (+ 117us) write_op.cc:270] Start()
0503 02:12:45.942268 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358425408 usec, L: 0
0503 02:12:45.942378 (+ 110us) write_op.cc:270] Start()
0503 02:12:45.942407 (+ 29us) write_op.cc:276] Timestamp: P: 1714702358426619 usec, L: 0
0503 02:12:45.942542 (+ 135us) write_op.cc:270] Start()
0503 02:12:45.942563 (+ 21us) write_op.cc:276] Timestamp: P: 1714702358455386 usec, L: 0
0503 02:12:45.942682 (+ 119us) write_op.cc:270] Start()
0503 02:12:45.942697 (+ 15us) write_op.cc:276] Timestamp: P: 1714702358456626 usec, L: 0
0503 02:12:45.942816 (+ 119us) write_op.cc:270] Start()
0503 02:12:45.942833 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358457708 usec, L: 0
0503 02:12:45.942955 (+ 122us) write_op.cc:270] Start()
0503 02:12:45.942972 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358459112 usec, L: 0
0503 02:12:45.943074 (+ 102us) write_op.cc:270] Start()
0503 02:12:45.943091 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358496956 usec, L: 0
0503 02:12:45.943199 (+ 108us) write_op.cc:270] Start()
0503 02:12:45.943219 (+ 20us) write_op.cc:276] Timestamp: P: 1714702358565281 usec, L: 0
0503 02:12:45.943333 (+ 114us) write_op.cc:270] Start()
0503 02:12:45.943350 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358583669 usec, L: 0
0503 02:12:45.943492 (+ 142us) write_op.cc:270] Start()
0503 02:12:45.943513 (+ 21us) write_op.cc:276] Timestamp: P: 1714702358594231 usec, L: 0
0503 02:12:45.943650 (+ 137us) write_op.cc:270] Start()
0503 02:12:45.943670 (+ 20us) write_op.cc:276] Timestamp: P: 1714702358594937 usec, L: 0
0503 02:12:45.943792 (+ 122us) write_op.cc:270] Start()
0503 02:12:45.943810 (+ 18us) write_op.cc:276] Timestamp: P: 1714702358595544 usec, L: 0
0503 02:12:45.943941 (+ 131us) write_op.cc:270] Start()
0503 02:12:45.943960 (+ 19us) write_op.cc:276] Timestamp: P: 1714702358609666 usec, L: 0
0503 02:12:45.944074 (+ 114us) write_op.cc:270] Start()
0503 02:12:45.944091 (+ 17us) write_op.cc:276] Timestamp: P: 1714702358711254 usec, L: 0
0503 02:12:45.944216 (+ 125us) write_op.cc:270] Start()
0503 02:12:45.944234 (+ 18us) write_op.cc:276] Timestamp: P: 1714702358717049 usec, L: 0
0503 02:12:45.944373 (+ 139us) write_op.cc:270] Start()
0503 02:12:45.944392 (+ 19us) write_op.cc:276] Timestamp: P: 1714702358718413 usec, L: 0
0503 02:12:46.048461 (+104069us) write_op.cc:270] Start()
0503 02:12:46.048518 (+ 57us) write_op.cc:276] Timestamp: P: 1714702358719732 usec, L: 0
0503 02:12:46.053075 (+ 4557us) write_op.cc:270] Start()
0503 02:12:46.053128 (+ 53us) write_op.cc:276] Timestamp: P: 1714702358979626 usec, L: 0
0503 02:12:46.067613 (+ 14485us) write_op.cc:270] Start()
0503 02:12:46.067666 (+ 53us) write_op.cc:276] Timestamp: P: 1714702358980963 usec, L: 0
0503 02:12:46.101664 (+ 33998us) write_op.cc:270] Start()
0503 02:12:46.101717 (+ 53us) write_op.cc:276] Timestamp: P: 1714702359160367 usec, L: 0
0503 02:12:46.129077 (+ 27360us) write_op.cc:270] Start()
0503 02:12:46.129145 (+ 68us) write_op.cc:276] Timestamp: P: 1714702359161889 usec, L: 0
0503 02:12:46.129433 (+ 288us) write_op.cc:270] Start()
0503 02:12:46.129461 (+ 28us) write_op.cc:276] Timestamp: P: 1714702359163101 usec, L: 0
0503 02:12:46.130237 (+ 776us) write_op.cc:270] Start()
0503 02:12:46.130267 (+ 30us) write_op.cc:276] Timestamp: P: 1714702359164835 usec, L: 0
0503 02:12:46.130447 (+ 180us) write_op.cc:270] Start()
0503 02:12:46.130472 (+ 25us) write_op.cc:276] Timestamp: P: 1714702359166946 usec, L: 0
0503 02:12:46.130616 (+ 144us) write_op.cc:270] Start()
0503 02:12:46.130637 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359242006 usec, L: 0
0503 02:12:46.130766 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.130785 (+ 19us) write_op.cc:276] Timestamp: P: 1714702359307638 usec, L: 0
0503 02:12:46.130914 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.130935 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359308624 usec, L: 0
0503 02:12:46.131064 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.131085 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359318654 usec, L: 0
0503 02:12:46.131224 (+ 139us) write_op.cc:270] Start()
0503 02:12:46.131247 (+ 23us) write_op.cc:276] Timestamp: P: 1714702359319625 usec, L: 0
0503 02:12:46.131406 (+ 159us) write_op.cc:270] Start()
0503 02:12:46.131461 (+ 55us) write_op.cc:276] Timestamp: P: 1714702359320412 usec, L: 0
0503 02:12:46.131616 (+ 155us) write_op.cc:270] Start()
0503 02:12:46.131638 (+ 22us) write_op.cc:276] Timestamp: P: 1714702359339805 usec, L: 0
0503 02:12:46.131768 (+ 130us) write_op.cc:270] Start()
0503 02:12:46.131789 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359346135 usec, L: 0
0503 02:12:46.131942 (+ 153us) write_op.cc:270] Start()
0503 02:12:46.131966 (+ 24us) write_op.cc:276] Timestamp: P: 1714702359348653 usec, L: 0
0503 02:12:46.132103 (+ 137us) write_op.cc:270] Start()
0503 02:12:46.132124 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359382250 usec, L: 0
0503 02:12:46.132294 (+ 170us) write_op.cc:270] Start()
0503 02:12:46.132316 (+ 22us) write_op.cc:276] Timestamp: P: 1714702359382867 usec, L: 0
0503 02:12:46.132478 (+ 162us) write_op.cc:270] Start()
0503 02:12:46.132499 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359394404 usec, L: 0
0503 02:12:46.149065 (+ 16566us) write_op.cc:270] Start()
0503 02:12:46.149133 (+ 68us) write_op.cc:276] Timestamp: P: 1714702359395195 usec, L: 0
0503 02:12:46.149422 (+ 289us) write_op.cc:270] Start()
0503 02:12:46.149451 (+ 29us) write_op.cc:276] Timestamp: P: 1714702359395619 usec, L: 0
0503 02:12:46.149602 (+ 151us) write_op.cc:270] Start()
0503 02:12:46.149623 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359401407 usec, L: 0
0503 02:12:46.149944 (+ 321us) write_op.cc:270] Start()
0503 02:12:46.149965 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359443314 usec, L: 0
0503 02:12:46.150093 (+ 128us) write_op.cc:270] Start()
0503 02:12:46.150116 (+ 23us) write_op.cc:276] Timestamp: P: 1714702359445765 usec, L: 0
0503 02:12:46.150312 (+ 196us) write_op.cc:270] Start()
0503 02:12:46.150338 (+ 26us) write_op.cc:276] Timestamp: P: 1714702359446455 usec, L: 0
0503 02:12:46.150490 (+ 152us) write_op.cc:270] Start()
0503 02:12:46.150510 (+ 20us) write_op.cc:276] Timestamp: P: 1714702359454622 usec, L: 0
0503 02:12:46.150642 (+ 132us) write_op.cc:270] Start()
0503 02:12:46.150659 (+ 17us) write_op.cc:276] Timestamp: P: 1714702359455294 usec, L: 0
0503 02:12:46.150788 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.150821 (+ 33us) write_op.cc:276] Timestamp: P: 1714702359477605 usec, L: 0
0503 02:12:46.150964 (+ 143us) write_op.cc:270] Start()
0503 02:12:46.150988 (+ 24us) write_op.cc:276] Timestamp: P: 1714702359518404 usec, L: 0
0503 02:12:46.151127 (+ 139us) write_op.cc:270] Start()
0503 02:12:46.151151 (+ 24us) write_op.cc:276] Timestamp: P: 1714702359524009 usec, L: 0
0503 02:12:46.151275 (+ 124us) write_op.cc:270] Start()
0503 02:12:46.151305 (+ 30us) write_op.cc:276] Timestamp: P: 1714702359528120 usec, L: 0
0503 02:12:46.151606 (+ 301us) write_op.cc:270] Start()
0503 02:12:46.151638 (+ 32us) write_op.cc:276] Timestamp: P: 1714702359528799 usec, L: 0
0503 02:12:46.151811 (+ 173us) write_op.cc:270] Start()
0503 02:12:46.151831 (+ 20us) write_op.cc:276] Timestamp: P: 1714702359534586 usec, L: 0
0503 02:12:46.151972 (+ 141us) write_op.cc:270] Start()
0503 02:12:46.151992 (+ 20us) write_op.cc:276] Timestamp: P: 1714702359617637 usec, L: 0
0503 02:12:46.152111 (+ 119us) write_op.cc:270] Start()
0503 02:12:46.152129 (+ 18us) write_op.cc:276] Timestamp: P: 1714702359644287 usec, L: 0
0503 02:12:46.152258 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.152278 (+ 20us) write_op.cc:276] Timestamp: P: 1714702359644867 usec, L: 0
0503 02:12:46.152411 (+ 133us) write_op.cc:270] Start()
0503 02:12:46.152429 (+ 18us) write_op.cc:276] Timestamp: P: 1714702359645414 usec, L: 0
0503 02:12:46.152548 (+ 119us) write_op.cc:270] Start()
0503 02:12:46.152564 (+ 16us) write_op.cc:276] Timestamp: P: 1714702359645922 usec, L: 0
0503 02:12:46.152664 (+ 100us) write_op.cc:270] Start()
0503 02:12:46.152680 (+ 16us) write_op.cc:276] Timestamp: P: 1714702359650592 usec, L: 0
0503 02:12:46.152788 (+ 108us) write_op.cc:270] Start()
0503 02:12:46.152804 (+ 16us) write_op.cc:276] Timestamp: P: 1714702359655707 usec, L: 0
0503 02:12:46.152904 (+ 100us) write_op.cc:270] Start()
0503 02:12:46.356329 (+203425us) write_op.cc:276] Timestamp: P: 1714702359676343 usec, L: 0
0503 02:12:46.356628 (+ 299us) write_op.cc:270] Start()
0503 02:12:46.356662 (+ 34us) write_op.cc:276] Timestamp: P: 1714702359685298 usec, L: 0
0503 02:12:46.356797 (+ 135us) write_op.cc:270] Start()
0503 02:12:46.356818 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359701947 usec, L: 0
0503 02:12:46.356981 (+ 163us) write_op.cc:270] Start()
0503 02:12:46.357002 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359702458 usec, L: 0
0503 02:12:46.357132 (+ 130us) write_op.cc:270] Start()
0503 02:12:46.357149 (+ 17us) write_op.cc:276] Timestamp: P: 1714702359726125 usec, L: 0
0503 02:12:46.357251 (+ 102us) write_op.cc:270] Start()
0503 02:12:46.357267 (+ 16us) write_op.cc:276] Timestamp: P: 1714702359816245 usec, L: 0
0503 02:12:46.357388 (+ 121us) write_op.cc:270] Start()
0503 02:12:46.357405 (+ 17us) write_op.cc:276] Timestamp: P: 1714702359823354 usec, L: 0
0503 02:12:46.357519 (+ 114us) write_op.cc:270] Start()
0503 02:12:46.357537 (+ 18us) write_op.cc:276] Timestamp: P: 1714702359867957 usec, L: 0
0503 02:12:46.357666 (+ 129us) write_op.cc:270] Start()
0503 02:12:46.357683 (+ 17us) write_op.cc:276] Timestamp: P: 1714702359868598 usec, L: 0
0503 02:12:46.357785 (+ 102us) write_op.cc:270] Start()
0503 02:12:46.357799 (+ 14us) write_op.cc:276] Timestamp: P: 1714702359869135 usec, L: 0
0503 02:12:46.357945 (+ 146us) write_op.cc:270] Start()
0503 02:12:46.357962 (+ 17us) write_op.cc:276] Timestamp: P: 1714702359869585 usec, L: 0
0503 02:12:46.358098 (+ 136us) write_op.cc:270] Start()
0503 02:12:46.358119 (+ 21us) write_op.cc:276] Timestamp: P: 1714702359870059 usec, L: 0
0503 02:12:46.358217 (+ 98us) write_op.cc:270] Start()
0503 02:12:46.358232 (+ 15us) write_op.cc:276] Timestamp: P: 1714702359872027 usec, L: 0
0503 02:12:46.358334 (+ 102us) write_op.cc:270] Start()
0503 02:12:46.358353 (+ 19us) write_op.cc:276] Timestamp: P: 1714702359878353 usec, L: 0
0503 02:12:46.358485 (+ 132us) write_op.cc:270] Start()
0503 02:12:46.358504 (+ 19us) write_op.cc:276] Timestamp: P: 1714702359895898 usec, L: 0
0503 02:12:46.358945 (+ 441us) write_op.cc:270] Start()
0503 02:12:46.358977 (+ 32us) write_op.cc:276] Timestamp: P: 1714702359897667 usec, L: 0
0503 02:12:46.359159 (+ 182us) write_op.cc:270] Start()
0503 02:12:46.359181 (+ 22us) write_op.cc:276] Timestamp: P: 1714702359899389 usec, L: 0
0503 02:12:46.359319 (+ 138us) write_op.cc:270] Start()
0503 02:12:46.359345 (+ 26us) write_op.cc:276] Timestamp: P: 1714702359916993 usec, L: 0
0503 02:12:46.359508 (+ 163us) write_op.cc:270] Start()
0503 02:12:46.359530 (+ 22us) write_op.cc:276] Timestamp: P: 1714702359919215 usec, L: 0
0503 02:12:46.359708 (+ 178us) write_op.cc:270] Start()
0503 02:12:46.418568 (+ 58860us) write_op.cc:276] Timestamp: P: 1714702359967122 usec, L: 0
0503 02:12:46.422947 (+ 4379us) write_op.cc:270] Start()
0503 02:12:46.422994 (+ 47us) write_op.cc:276] Timestamp: P: 1714702359984160 usec, L: 0
0503 02:12:46.423232 (+ 238us) write_op.cc:270] Start()
0503 02:12:46.423258 (+ 26us) write_op.cc:276] Timestamp: P: 1714702360005526 usec, L: 0
0503 02:12:46.423427 (+ 169us) write_op.cc:270] Start()
0503 02:12:46.423463 (+ 36us) write_op.cc:276] Timestamp: P: 1714702360006550 usec, L: 0
0503 02:12:46.423615 (+ 152us) write_op.cc:270] Start()
0503 02:12:46.423637 (+ 22us) write_op.cc:276] Timestamp: P: 1714702360007632 usec, L: 0
0503 02:12:46.423979 (+ 342us) write_op.cc:270] Start()
0503 02:12:46.424007 (+ 28us) write_op.cc:276] Timestamp: P: 1714702360008671 usec, L: 0
0503 02:12:46.424185 (+ 178us) write_op.cc:270] Start()
0503 02:12:46.424208 (+ 23us) write_op.cc:276] Timestamp: P: 1714702360027815 usec, L: 0
0503 02:12:46.424378 (+ 170us) write_op.cc:270] Start()
0503 02:12:46.424401 (+ 23us) write_op.cc:276] Timestamp: P: 1714702360038822 usec, L: 0
0503 02:12:46.424580 (+ 179us) write_op.cc:270] Start()
0503 02:12:46.424608 (+ 28us) write_op.cc:276] Timestamp: P: 1714702360051042 usec, L: 0
0503 02:12:46.424754 (+ 146us) write_op.cc:270] Start()
0503 02:12:46.424773 (+ 19us) write_op.cc:276] Timestamp: P: 1714702360079909 usec, L: 0
0503 02:12:46.424908 (+ 135us) write_op.cc:270] Start()
0503 02:12:46.424991 (+ 83us) write_op.cc:276] Timestamp: P: 1714702360360363 usec, L: 0
0503 02:12:46.425119 (+ 128us) write_op.cc:270] Start()
0503 02:12:46.425136 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360366161 usec, L: 0
0503 02:12:46.425247 (+ 111us) write_op.cc:270] Start()
0503 02:12:46.425264 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360373629 usec, L: 0
0503 02:12:46.425365 (+ 101us) write_op.cc:270] Start()
0503 02:12:46.425380 (+ 15us) write_op.cc:276] Timestamp: P: 1714702360374301 usec, L: 0
0503 02:12:46.425490 (+ 110us) write_op.cc:270] Start()
0503 02:12:46.425508 (+ 18us) write_op.cc:276] Timestamp: P: 1714702360374924 usec, L: 0
0503 02:12:46.425626 (+ 118us) write_op.cc:270] Start()
0503 02:12:46.425643 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360377589 usec, L: 0
0503 02:12:46.425774 (+ 131us) write_op.cc:270] Start()
0503 02:12:46.425791 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360378190 usec, L: 0
0503 02:12:46.425892 (+ 101us) write_op.cc:270] Start()
0503 02:12:46.425909 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360379416 usec, L: 0
0503 02:12:46.426090 (+ 181us) write_op.cc:270] Start()
0503 02:12:46.426111 (+ 21us) write_op.cc:276] Timestamp: P: 1714702360416293 usec, L: 0
0503 02:12:46.426253 (+ 142us) write_op.cc:270] Start()
0503 02:12:46.426285 (+ 32us) write_op.cc:276] Timestamp: P: 1714702360416994 usec, L: 0
0503 02:12:46.426400 (+ 115us) write_op.cc:270] Start()
0503 02:12:46.426420 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360417995 usec, L: 0
0503 02:12:46.426547 (+ 127us) write_op.cc:270] Start()
0503 02:12:46.426566 (+ 19us) write_op.cc:276] Timestamp: P: 1714702360418545 usec, L: 0
0503 02:12:46.426699 (+ 133us) write_op.cc:270] Start()
0503 02:12:46.426720 (+ 21us) write_op.cc:276] Timestamp: P: 1714702360419026 usec, L: 0
0503 02:12:46.426833 (+ 113us) write_op.cc:270] Start()
0503 02:12:46.426851 (+ 18us) write_op.cc:276] Timestamp: P: 1714702360607621 usec, L: 0
0503 02:12:46.426986 (+ 135us) write_op.cc:270] Start()
0503 02:12:46.427028 (+ 42us) write_op.cc:276] Timestamp: P: 1714702360608419 usec, L: 0
0503 02:12:46.427150 (+ 122us) write_op.cc:270] Start()
0503 02:12:46.427169 (+ 19us) write_op.cc:276] Timestamp: P: 1714702360609094 usec, L: 0
0503 02:12:46.427286 (+ 117us) write_op.cc:270] Start()
0503 02:12:46.427304 (+ 18us) write_op.cc:276] Timestamp: P: 1714702360609630 usec, L: 0
0503 02:12:46.427406 (+ 102us) write_op.cc:270] Start()
0503 02:12:46.427423 (+ 17us) write_op.cc:276] Timestamp: P: 1714702360610096 usec, L: 0
0503 02:12:46.513335 (+ 85912us) write_op.cc:270] Start()
0503 02:12:46.513390 (+ 55us) write_op.cc:276] Timestamp: P: 1714702360610553 usec, L: 0
0503 02:12:46.513678 (+ 288us) write_op.cc:270] Start()
0503 02:12:46.513712 (+ 34us) write_op.cc:276] Timestamp: P: 1714702360635674 usec, L: 0
0503 02:12:46.513866 (+ 154us) write_op.cc:270] Start()
0503 02:12:46.513886 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360637295 usec, L: 0
0503 02:12:46.513993 (+ 107us) write_op.cc:270] Start()
0503 02:12:46.514009 (+ 16us) write_op.cc:276] Timestamp: P: 1714702360638645 usec, L: 0
0503 02:12:46.514110 (+ 101us) write_op.cc:270] Start()
0503 02:12:46.514126 (+ 16us) write_op.cc:276] Timestamp: P: 1714702360640227 usec, L: 0
0503 02:12:46.514219 (+ 93us) write_op.cc:270] Start()
0503 02:12:46.514235 (+ 16us) write_op.cc:276] Timestamp: P: 1714702360641878 usec, L: 0
0503 02:12:46.514357 (+ 122us) write_op.cc:270] Start()
0503 02:12:46.514371 (+ 14us) write_op.cc:276] Timestamp: P: 1714702360643460 usec, L: 0
0503 02:12:46.514516 (+ 145us) write_op.cc:270] Start()
0503 02:12:46.514540 (+ 24us) write_op.cc:276] Timestamp: P: 1714702360647504 usec, L: 0
0503 02:12:46.514690 (+ 150us) write_op.cc:270] Start()
0503 02:12:46.514710 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360697005 usec, L: 0
0503 02:12:46.514857 (+ 147us) write_op.cc:270] Start()
0503 02:12:46.514877 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360697736 usec, L: 0
0503 02:12:46.514995 (+ 118us) write_op.cc:270] Start()
0503 02:12:46.515014 (+ 19us) write_op.cc:276] Timestamp: P: 1714702360704204 usec, L: 0
0503 02:12:46.515129 (+ 115us) write_op.cc:270] Start()
0503 02:12:46.515147 (+ 18us) write_op.cc:276] Timestamp: P: 1714702360704811 usec, L: 0
0503 02:12:46.515262 (+ 115us) write_op.cc:270] Start()
0503 02:12:46.515319 (+ 57us) write_op.cc:276] Timestamp: P: 1714702360715668 usec, L: 0
0503 02:12:46.515458 (+ 139us) write_op.cc:270] Start()
0503 02:12:46.515478 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360720512 usec, L: 0
0503 02:12:46.515643 (+ 165us) write_op.cc:270] Start()
0503 02:12:46.515667 (+ 24us) write_op.cc:276] Timestamp: P: 1714702360733546 usec, L: 0
0503 02:12:46.515808 (+ 141us) write_op.cc:270] Start()
0503 02:12:46.515829 (+ 21us) write_op.cc:276] Timestamp: P: 1714702360742869 usec, L: 0
0503 02:12:46.515955 (+ 126us) write_op.cc:270] Start()
0503 02:12:46.515977 (+ 22us) write_op.cc:276] Timestamp: P: 1714702360748013 usec, L: 0
0503 02:12:46.516103 (+ 126us) write_op.cc:270] Start()
0503 02:12:46.516118 (+ 15us) write_op.cc:276] Timestamp: P: 1714702360819052 usec, L: 0
0503 02:12:46.543147 (+ 27029us) write_op.cc:270] Start()
0503 02:12:46.543200 (+ 53us) write_op.cc:276] Timestamp: P: 1714702360834585 usec, L: 0
0503 02:12:46.543430 (+ 230us) write_op.cc:270] Start()
0503 02:12:46.543459 (+ 29us) write_op.cc:276] Timestamp: P: 1714702360837718 usec, L: 0
0503 02:12:46.543577 (+ 118us) write_op.cc:270] Start()
0503 02:12:46.543593 (+ 16us) write_op.cc:276] Timestamp: P: 1714702360838657 usec, L: 0
0503 02:12:46.543716 (+ 123us) write_op.cc:270] Start()
0503 02:12:46.543735 (+ 19us) write_op.cc:276] Timestamp: P: 1714702360839474 usec, L: 0
0503 02:12:46.543875 (+ 140us) write_op.cc:270] Start()
0503 02:12:46.543898 (+ 23us) write_op.cc:276] Timestamp: P: 1714702360840347 usec, L: 0
0503 02:12:46.544043 (+ 145us) write_op.cc:270] Start()
0503 02:12:46.544063 (+ 20us) write_op.cc:276] Timestamp: P: 1714702360841426 usec, L: 0
0503 02:12:46.544173 (+ 110us) write_op.cc:270] Start()
0503 02:12:46.544189 (+ 16us) write_op.cc:276] Timestamp: P: 1714702360961985 usec, L: 0
0503 02:12:46.544311 (+ 122us) write_op.cc:270] Start()
0503 02:12:46.561031 (+ 16720us) write_op.cc:276] Timestamp: P: 1714702360964343 usec, L: 0
0503 02:12:46.561363 (+ 332us) write_op.cc:270] Start()
0503 02:12:46.561392 (+ 29us) write_op.cc:276] Timestamp: P: 1714702361090124 usec, L: 0
0503 02:12:46.561554 (+ 162us) write_op.cc:270] Start()
0503 02:12:46.561575 (+ 21us) write_op.cc:276] Timestamp: P: 1714702361090777 usec, L: 0
0503 02:12:46.561718 (+ 143us) write_op.cc:270] Start()
0503 02:12:46.561763 (+ 45us) write_op.cc:276] Timestamp: P: 1714702361091321 usec, L: 0
0503 02:12:46.561891 (+ 128us) write_op.cc:270] Start()
0503 02:12:46.561907 (+ 16us) write_op.cc:276] Timestamp: P: 1714702361099403 usec, L: 0
0503 02:12:46.606434 (+ 44527us) write_op.cc:270] Start()
0503 02:12:46.606496 (+ 62us) write_op.cc:276] Timestamp: P: 1714702361105297 usec, L: 0
0503 02:12:46.606765 (+ 269us) write_op.cc:270] Start()
0503 02:12:46.606787 (+ 22us) write_op.cc:276] Timestamp: P: 1714702361107692 usec, L: 0
0503 02:12:46.606953 (+ 166us) write_op.cc:270] Start()
0503 02:12:46.606977 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361132900 usec, L: 0
0503 02:12:46.607131 (+ 154us) write_op.cc:270] Start()
0503 02:12:46.607151 (+ 20us) write_op.cc:276] Timestamp: P: 1714702361134252 usec, L: 0
0503 02:12:46.607282 (+ 131us) write_op.cc:270] Start()
0503 02:12:46.607303 (+ 21us) write_op.cc:276] Timestamp: P: 1714702361134904 usec, L: 0
0503 02:12:46.607422 (+ 119us) write_op.cc:270] Start()
0503 02:12:46.607451 (+ 29us) write_op.cc:276] Timestamp: P: 1714702361135449 usec, L: 0
0503 02:12:46.607583 (+ 132us) write_op.cc:270] Start()
0503 02:12:46.607601 (+ 18us) write_op.cc:276] Timestamp: P: 1714702361179843 usec, L: 0
0503 02:12:46.607727 (+ 126us) write_op.cc:270] Start()
0503 02:12:46.607747 (+ 20us) write_op.cc:276] Timestamp: P: 1714702361194529 usec, L: 0
0503 02:12:46.607890 (+ 143us) write_op.cc:270] Start()
0503 02:12:46.607913 (+ 23us) write_op.cc:276] Timestamp: P: 1714702361198080 usec, L: 0
0503 02:12:46.608069 (+ 156us) write_op.cc:270] Start()
0503 02:12:46.608087 (+ 18us) write_op.cc:276] Timestamp: P: 1714702361198745 usec, L: 0
0503 02:12:46.608207 (+ 120us) write_op.cc:270] Start()
0503 02:12:46.608223 (+ 16us) write_op.cc:276] Timestamp: P: 1714702361199319 usec, L: 0
0503 02:12:46.608327 (+ 104us) write_op.cc:270] Start()
0503 02:12:46.608342 (+ 15us) write_op.cc:276] Timestamp: P: 1714702361199858 usec, L: 0
0503 02:12:46.622333 (+ 13991us) write_op.cc:270] Start()
0503 02:12:46.622380 (+ 47us) write_op.cc:276] Timestamp: P: 1714702361201317 usec, L: 0
0503 02:12:46.622668 (+ 288us) write_op.cc:270] Start()
0503 02:12:46.622700 (+ 32us) write_op.cc:276] Timestamp: P: 1714702361301942 usec, L: 0
0503 02:12:46.622878 (+ 178us) write_op.cc:270] Start()
0503 02:12:46.622903 (+ 25us) write_op.cc:276] Timestamp: P: 1714702361302542 usec, L: 0
0503 02:12:46.623128 (+ 225us) write_op.cc:270] Start()
0503 02:12:46.623154 (+ 26us) write_op.cc:276] Timestamp: P: 1714702361303056 usec, L: 0
0503 02:12:46.624568 (+ 1414us) write_op.cc:270] Start()
0503 02:12:46.624628 (+ 60us) write_op.cc:276] Timestamp: P: 1714702361303565 usec, L: 0
0503 02:12:46.624901 (+ 273us) write_op.cc:270] Start()
0503 02:12:46.624951 (+ 50us) write_op.cc:276] Timestamp: P: 1714702361314316 usec, L: 0
0503 02:12:46.625138 (+ 187us) write_op.cc:270] Start()
0503 02:12:46.625166 (+ 28us) write_op.cc:276] Timestamp: P: 1714702361315534 usec, L: 0
0503 02:12:46.625320 (+ 154us) write_op.cc:270] Start()
0503 02:12:46.625344 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361316139 usec, L: 0
0503 02:12:46.625510 (+ 166us) write_op.cc:270] Start()
0503 02:12:46.625534 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361316650 usec, L: 0
0503 02:12:46.625675 (+ 141us) write_op.cc:270] Start()
0503 02:12:46.625700 (+ 25us) write_op.cc:276] Timestamp: P: 1714702361320256 usec, L: 0
0503 02:12:46.625855 (+ 155us) write_op.cc:270] Start()
0503 02:12:46.625881 (+ 26us) write_op.cc:276] Timestamp: P: 1714702361323605 usec, L: 0
0503 02:12:46.625989 (+ 108us) write_op.cc:270] Start()
0503 02:12:46.626012 (+ 23us) write_op.cc:276] Timestamp: P: 1714702361553337 usec, L: 0
0503 02:12:46.626208 (+ 196us) write_op.cc:270] Start()
0503 02:12:46.626235 (+ 27us) write_op.cc:276] Timestamp: P: 1714702361557269 usec, L: 0
0503 02:12:46.642421 (+ 16186us) write_op.cc:270] Start()
0503 02:12:46.642479 (+ 58us) write_op.cc:276] Timestamp: P: 1714702361574329 usec, L: 0
0503 02:12:46.642742 (+ 263us) write_op.cc:270] Start()
0503 02:12:46.642772 (+ 30us) write_op.cc:276] Timestamp: P: 1714702361574948 usec, L: 0
0503 02:12:46.642919 (+ 147us) write_op.cc:270] Start()
0503 02:12:46.642943 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361575534 usec, L: 0
0503 02:12:46.643076 (+ 133us) write_op.cc:270] Start()
0503 02:12:46.643096 (+ 20us) write_op.cc:276] Timestamp: P: 1714702361576103 usec, L: 0
0503 02:12:46.643235 (+ 139us) write_op.cc:270] Start()
0503 02:12:46.643259 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361591975 usec, L: 0
0503 02:12:46.643428 (+ 169us) write_op.cc:270] Start()
0503 02:12:46.643462 (+ 34us) write_op.cc:276] Timestamp: P: 1714702361592623 usec, L: 0
0503 02:12:46.643589 (+ 127us) write_op.cc:270] Start()
0503 02:12:46.643613 (+ 24us) write_op.cc:276] Timestamp: P: 1714702361598387 usec, L: 0
0503 02:12:46.643769 (+ 156us) write_op.cc:270] Start()
0503 02:12:46.643788 (+ 19us) write_op.cc:276] Timestamp: P: 171470
I20240503 02:12:48.380630 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8287b45fa0b492ca82c22164f3a7837" candidate_uuid: "1f50719d94564844b15e1a9211d48a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc6d33969e74c209310005ae911c405"
I20240503 02:12:48.380990 4805 raft_consensus.cc:3037] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.383934 4805 raft_consensus.cc:2454] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f50719d94564844b15e1a9211d48a81 in term 1.
I20240503 02:12:48.386893 7003 raft_consensus.cc:482] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.387085 7003 raft_consensus.cc:504] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.387745 7003 leader_election.cc:288] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.393621 5105 tablet_bootstrap.cc:492] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.393797 5105 ts_tablet_manager.cc:1367] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.003s
I20240503 02:12:48.394572 5105 raft_consensus.cc:348] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.394801 5105 raft_consensus.cc:374] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.394876 5105 raft_consensus.cc:729] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.385061 5106 tablet_bootstrap.cc:492] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.389061 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890fb92ca2314f2e81e1ffb0bcddb0b1" candidate_uuid: "acc6d33969e74c209310005ae911c405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f50719d94564844b15e1a9211d48a81" is_pre_election: true
I20240503 02:12:48.395171 5106 ts_tablet_manager.cc:1367] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.005s
I20240503 02:12:48.392163 4608 leader_election.cc:302] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f50719d94564844b15e1a9211d48a81, acc6d33969e74c209310005ae911c405; no voters:
W20240503 02:12:48.395905 4608 consensus_peers.cc:485] T dee34be89bda450296b916163bb10c46 P 1f50719d94564844b15e1a9211d48a81 -> Peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Couldn't send request to peer acc6d33969e74c209310005ae911c405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dee34be89bda450296b916163bb10c46. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.396111 4746 leader_election.cc:341] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:48.396322 7019 raft_consensus.cc:2781] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.396454 7019 raft_consensus.cc:686] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Running, Role: LEADER
I20240503 02:12:48.396674 7019 consensus_queue.cc:237] T b8287b45fa0b492ca82c22164f3a7837 P 1f50719d94564844b15e1a9211d48a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.395932 5106 raft_consensus.cc:348] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.398259 5106 raft_consensus.cc:374] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.398370 5106 raft_consensus.cc:729] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.398798 5106 consensus_queue.cc:260] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.399798 5106 ts_tablet_manager.cc:1398] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:48.400583 5106 tablet_bootstrap.cc:492] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.402601 5106 tablet_bootstrap.cc:654] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.403921 5105 consensus_queue.cc:260] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.405139 5105 ts_tablet_manager.cc:1398] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:12:48.407052 5105 tablet_bootstrap.cc:492] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.413745 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
I20240503 02:12:48.414496 5105 tablet_bootstrap.cc:654] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.416321 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1092) took 1004 ms. Trace:
I20240503 02:12:48.416852 5159 rpcz_store.cc:276] 0503 02:12:47.412038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.412362 (+ 324us) service_pool.cc:225] Handling call
0503 02:12:48.315284 (+902922us) ts_tablet_manager.cc:1120] Deleting tablet 5cd296bb3b574d6cbb0252ad10ada00c
0503 02:12:48.317693 (+ 2409us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.324447 (+ 6754us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.343984 (+ 19537us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.416258 (+ 72274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":902459}
I20240503 02:12:48.418081 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 5cd296bb3b574d6cbb0252ad10ada00c (table default.Table_9947777ee256419394d206f28a1a6e11 [id=f7f88566af964cefb2026fb433ca0f91]) successfully deleted
I20240503 02:12:48.422091 5159 tablet_replica.cc:330] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.422778 5159 raft_consensus.cc:2229] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.425355 5159 raft_consensus.cc:2258] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.427424 5159 ts_tablet_manager.cc:1875] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.429065 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b6a18be64049049e1a083bf4af51ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:57782
I20240503 02:12:48.431707 5105 tablet_bootstrap.cc:492] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.432009 7020 raft_consensus.cc:482] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.432161 7020 raft_consensus.cc:504] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.432931 7020 leader_election.cc:288] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:48.434075 4608 leader_election.cc:341] T 3478033c4ed94268a2436a2942c27a68 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: 3478033c4ed94268a2436a2942c27a68
I20240503 02:12:48.435189 5106 tablet_bootstrap.cc:492] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.435329 5106 ts_tablet_manager.cc:1367] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20240503 02:12:48.436084 5106 raft_consensus.cc:348] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.436266 5106 raft_consensus.cc:374] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.436317 5106 raft_consensus.cc:729] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.437906 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
I20240503 02:12:48.438447 5105 ts_tablet_manager.cc:1367] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 02:12:48.439972 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet a88710ac9062472f80bf6fcfb82793a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.443989 5106 consensus_queue.cc:260] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.444293 5105 raft_consensus.cc:348] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.444527 5105 raft_consensus.cc:374] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.444605 5105 raft_consensus.cc:729] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.445052 5115 ts_tablet_manager.cc:1888] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 02:12:48.446277 5159 ts_tablet_manager.cc:1888] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.446306 5115 log.cc:1196] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/79aae500e9a842ed90357f4a65fd37ba
I20240503 02:12:48.446419 5159 log.cc:1196] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a8ce4b33f90c40c48cf0a538a1a7fde4
I20240503 02:12:48.446774 5159 ts_tablet_manager.cc:1909] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.449604 5106 ts_tablet_manager.cc:1398] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 02:12:48.449918 5106 tablet_bootstrap.cc:492] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.450017 5115 ts_tablet_manager.cc:1909] T 79aae500e9a842ed90357f4a65fd37ba P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
I20240503 02:12:48.452040 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 975) took 3394 ms. Trace:
I20240503 02:12:48.457013 5115 rpcz_store.cc:276] 0503 02:12:45.057306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.058397 (+ 1091us) service_pool.cc:225] Handling call
0503 02:12:48.369573 (+3311176us) ts_tablet_manager.cc:1120] Deleting tablet 79aae500e9a842ed90357f4a65fd37ba
0503 02:12:48.369580 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.402076 (+ 32496us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.444935 (+ 42859us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.452015 (+ 7080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3310980}
I20240503 02:12:48.458299 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 79aae500e9a842ed90357f4a65fd37ba (table default.Table_83329138bcd24006aa233387b43098a7 [id=e3d639800758474aaeef718686d0aa1a]) successfully deleted
I20240503 02:12:48.458675 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 997) took 3038 ms. Trace:
I20240503 02:12:48.464993 5115 rpcz_store.cc:276] 0503 02:12:45.419763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.419809 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:48.458497 (+3038688us) ts_tablet_manager.cc:1120] Deleting tablet db6fa251094842b9a629fdb67e20690b
0503 02:12:48.458510 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.458655 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2946272}
W20240503 02:12:48.467271 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet db6fa251094842b9a629fdb67e20690b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db6fa251094842b9a629fdb67e20690b already in progress: creating tablet
I20240503 02:12:48.467360 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 998) took 3047 ms. Trace:
I20240503 02:12:48.467447 5115 rpcz_store.cc:276] 0503 02:12:45.420049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.420094 (+ 45us) service_pool.cc:225] Handling call
0503 02:12:48.467212 (+3047118us) ts_tablet_manager.cc:1120] Deleting tablet ffef15b7e51840fea91b5349a1724cc5
0503 02:12:48.467225 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.467338 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954877}
W20240503 02:12:48.469419 4259 catalog_manager.cc:4810] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): delete failed for tablet ffef15b7e51840fea91b5349a1724cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffef15b7e51840fea91b5349a1724cc5 already in progress: creating tablet
I20240503 02:12:48.469913 5115 tablet_replica.cc:330] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.451558 5105 consensus_queue.cc:260] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.457013 5106 tablet_bootstrap.cc:654] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.471026 7019 raft_consensus.cc:482] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.471453 7019 raft_consensus.cc:504] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.472065 7019 leader_election.cc:288] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:48.472301 5115 raft_consensus.cc:2229] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.472482 5115 raft_consensus.cc:2258] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.473060 5115 ts_tablet_manager.cc:1875] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:48.473632 4608 leader_election.cc:341] T a8ce4b33f90c40c48cf0a538a1a7fde4 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): Not found: Tablet not found: a8ce4b33f90c40c48cf0a538a1a7fde4
I20240503 02:12:48.479212 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2850c07d042a7951a5fbeb0e4945d with delete_type TABLET_DATA_DELETED (replaced by tablet b2560f50415444b8b2e497453a558d3a) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.488548 5105 ts_tablet_manager.cc:1398] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240503 02:12:48.489923 5105 tablet_bootstrap.cc:492] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.504781 5105 tablet_bootstrap.cc:654] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.447243 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1094) took 1034 ms. Trace:
I20240503 02:12:48.506603 5159 rpcz_store.cc:276] 0503 02:12:47.412804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.412852 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:48.421069 (+1008217us) ts_tablet_manager.cc:1120] Deleting tablet a8ce4b33f90c40c48cf0a538a1a7fde4
0503 02:12:48.422015 (+ 946us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.442431 (+ 20416us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.446166 (+ 3735us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.447205 (+ 1039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008019}
I20240503 02:12:48.507714 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a8ce4b33f90c40c48cf0a538a1a7fde4 (table default.Table_f1c97fd1492d442e955a652306d0e6a6 [id=68bb420b28734c66bde996fa718da52e]) successfully deleted
I20240503 02:12:48.508173 5159 tablet_replica.cc:330] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.508472 5159 raft_consensus.cc:2229] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.509135 5159 raft_consensus.cc:2258] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.509783 5115 ts_tablet_manager.cc:1888] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:12:48.510875 5159 ts_tablet_manager.cc:1875] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.513092 5115 log.cc:1196] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/607e49cde2fd4aab9f44921d3c3921a5
I20240503 02:12:48.514950 5115 ts_tablet_manager.cc:1909] T 607e49cde2fd4aab9f44921d3c3921a5 P 1f50719d94564844b15e1a9211d48a81: Deleting consensus metadata
W20240503 02:12:48.518790 6192 connection.cc:535] server connection from 127.4.17.65:46199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:48.519694 5106 tablet_bootstrap.cc:492] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.522774 5106 ts_tablet_manager.cc:1367] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.006s
I20240503 02:12:48.523685 5106 raft_consensus.cc:348] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.523981 5106 raft_consensus.cc:374] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.524099 5106 raft_consensus.cc:729] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.524396 5106 consensus_queue.cc:260] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.525266 5106 ts_tablet_manager.cc:1398] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:48.525674 5106 tablet_bootstrap.cc:492] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.527632 5106 tablet_bootstrap.cc:654] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.532037 5105 tablet_bootstrap.cc:492] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.532212 5105 ts_tablet_manager.cc:1367] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.004s
I20240503 02:12:48.533085 5105 raft_consensus.cc:348] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.533376 5105 raft_consensus.cc:374] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.533473 5105 raft_consensus.cc:729] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.533743 5105 consensus_queue.cc:260] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.535847 5115 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57782 (request call id 999) took 3008 ms. Trace:
I20240503 02:12:48.539140 5115 rpcz_store.cc:276] 0503 02:12:45.527384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:45.527448 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:48.469856 (+2942408us) ts_tablet_manager.cc:1120] Deleting tablet 607e49cde2fd4aab9f44921d3c3921a5
0503 02:12:48.469868 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.476237 (+ 6369us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.509642 (+ 33405us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.535812 (+ 26170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2942199}
I20240503 02:12:48.539793 5115 tablet_replica.cc:330] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: stopping tablet replica
I20240503 02:12:48.540232 4259 catalog_manager.cc:4830] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): tablet 607e49cde2fd4aab9f44921d3c3921a5 (table default.Table_0e5ff92986ff447dbc2c9e69a628e8fe [id=08bfe029472248d7b17aaa9a8ee63472]) successfully deleted
I20240503 02:12:48.540871 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8b6599206549dd8ddf449a52c61fce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:37 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.544296 5115 raft_consensus.cc:2229] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.544515 5115 raft_consensus.cc:2258] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.548063 5105 ts_tablet_manager.cc:1398] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:12:48.551035 5115 ts_tablet_manager.cc:1875] T 27c13f257eae44689af07cc31ea254c4 P 1f50719d94564844b15e1a9211d48a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.554396 5106 tablet_bootstrap.cc:492] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:48.554559 5106 ts_tablet_manager.cc:1367] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20240503 02:12:48.558081 5106 raft_consensus.cc:348] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.558326 5106 raft_consensus.cc:374] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.558374 5106 raft_consensus.cc:729] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.558574 5106 consensus_queue.cc:260] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.559135 5106 ts_tablet_manager.cc:1398] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 02:12:48.561985 5106 tablet_bootstrap.cc:492] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
I20240503 02:12:48.562289 5105 tablet_bootstrap.cc:492] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.563728 5106 tablet_bootstrap.cc:654] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:48.565160 4608 connection.cc:535] client connection to 127.4.17.65:39837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:48.565320 4746 connection.cc:535] client connection to 127.4.17.65:39837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:48.565372 4259 connection.cc:535] client connection to 127.4.17.65:39837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:48.565423 4608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:48.565526 4259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 10 similar messages]
W20240503 02:12:48.565543 4746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:48.566952 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4585
I20240503 02:12:48.568331 4715 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:48.576774 5105 tablet_bootstrap.cc:654] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.578238 5159 ts_tablet_manager.cc:1888] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.578397 5159 log.cc:1196] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b8287b45fa0b492ca82c22164f3a7837
I20240503 02:12:48.578795 5159 ts_tablet_manager.cc:1909] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.579291 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1096) took 1084 ms. Trace:
I20240503 02:12:48.579414 5159 rpcz_store.cc:276] 0503 02:12:47.494438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.495089 (+ 651us) service_pool.cc:225] Handling call
0503 02:12:48.508092 (+1013003us) ts_tablet_manager.cc:1120] Deleting tablet b8287b45fa0b492ca82c22164f3a7837
0503 02:12:48.508105 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.526315 (+ 18210us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.578074 (+ 51759us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.579244 (+ 1170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1012603}
W20240503 02:12:48.589362 4746 leader_election.cc:334] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.589541 5159 tablet_replica.cc:330] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.589622 4746 leader_election.cc:302] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.589924 7003 raft_consensus.cc:2726] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.590049 4746 leader_election.cc:334] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.590193 4746 leader_election.cc:302] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.590437 7003 raft_consensus.cc:2726] T 890fb92ca2314f2e81e1ffb0bcddb0b1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:48.591317 4268 catalog_manager.cc:5576] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355) does not have the latest schema for tablet 83a1d5b2de8d4d3ba5190750545c9f7b (table default.Table_54f7e670055747f7862a072acefa597e [id=49901f1280764c1c8044fb4798a5c346]). Expected version 1 got 0
I20240503 02:12:48.589778 5159 raft_consensus.cc:2229] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.594722 5159 raft_consensus.cc:2258] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.595432 5159 ts_tablet_manager.cc:1875] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:48.596552 4746 consensus_peers.cc:485] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.599264 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b8287b45fa0b492ca82c22164f3a7837 (table default.Table_ef5ce757a4e349cab1cdea6c1bc22736 [id=7b088fcf3c334af29998d31075e5367b]) successfully deleted
W20240503 02:12:48.599886 4259 catalog_manager.cc:4573] TS f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4cb6432296c4a45850a96c3651ab7bd: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.597700 5105 tablet_bootstrap.cc:492] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:48.604115 4746 leader_election.cc:334] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.604797 4746 leader_election.cc:302] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.610543 7003 raft_consensus.cc:2726] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.611224 4746 leader_election.cc:334] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.611678 4746 leader_election.cc:302] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.612085 7003 raft_consensus.cc:2726] T 038f11066dc34ba9a833f84a54ff20bf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.614091 4746 leader_election.cc:334] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.614356 4746 leader_election.cc:302] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.614575 4746 raft_consensus.cc:2670] T aa699a1dc3c34be59925560d5e11f49e P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.615672 4746 leader_election.cc:334] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.616495 4746 leader_election.cc:302] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.618885 5159 ts_tablet_manager.cc:1888] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.619096 7003 raft_consensus.cc:2726] T 442d795ee01c417dbdeda41fd6853dd7 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.619336 4746 leader_election.cc:334] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.625032 4746 leader_election.cc:302] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.625322 4746 raft_consensus.cc:2670] T a8ce4b33f90c40c48cf0a538a1a7fde4 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.626055 4746 leader_election.cc:334] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.626258 4746 leader_election.cc:302] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.626473 4746 raft_consensus.cc:2670] T b8474c841cb64373a223ad76b43ecf5f P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.627056 4746 leader_election.cc:334] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.627251 4746 leader_election.cc:302] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.627478 5159 log.cc:1196] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/c3710b653aa24b7ca389aa95d7cdd2fe
I20240503 02:12:48.628000 5159 ts_tablet_manager.cc:1909] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.628163 7003 raft_consensus.cc:2726] T 54959b3e30944fab8bd5fd47ce42d246 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.628376 4746 leader_election.cc:334] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.628731 4746 leader_election.cc:302] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.629458 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1097) took 1066 ms. Trace:
I20240503 02:12:48.629642 7003 raft_consensus.cc:2726] T 6a11c306232c4e49ab02be0f195df935 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.629865 4746 consensus_peers.cc:485] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.630237 4746 leader_election.cc:334] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.630414 4746 leader_election.cc:302] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.630872 4746 raft_consensus.cc:2670] T 1ce9d746acfe4e9aaea600293470d373 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.631490 4746 leader_election.cc:334] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.631661 4746 leader_election.cc:302] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.631896 4746 raft_consensus.cc:2670] T b8287b45fa0b492ca82c22164f3a7837 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.632427 4746 leader_election.cc:334] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.632587 4746 leader_election.cc:302] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.632788 4746 raft_consensus.cc:2670] T c3710b653aa24b7ca389aa95d7cdd2fe P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.633646 4746 leader_election.cc:334] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.633819 4746 leader_election.cc:302] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.630573 5159 rpcz_store.cc:276] 0503 02:12:47.563415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.563487 (+ 72us) service_pool.cc:225] Handling call
0503 02:12:48.589461 (+1025974us) ts_tablet_manager.cc:1120] Deleting tablet c3710b653aa24b7ca389aa95d7cdd2fe
0503 02:12:48.589475 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.600144 (+ 10669us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.618747 (+ 18603us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.629435 (+ 10688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025709}
I20240503 02:12:48.634433 5159 tablet_replica.cc:330] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.634668 5159 raft_consensus.cc:2229] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.634842 7003 raft_consensus.cc:2726] T a86ac17982324847949b8a9245329489 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:48.635069 5159 raft_consensus.cc:2258] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.635646 5159 ts_tablet_manager.cc:1875] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.641242 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet c3710b653aa24b7ca389aa95d7cdd2fe (table default.Table_d6e976e60b3e445492636b06856344c3 [id=0f32eabc16cd47bdb6addf705adfffa7]) successfully deleted
I20240503 02:12:48.642107 5159 ts_tablet_manager.cc:1888] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.642244 5159 log.cc:1196] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/47c7ba3639bd4cb4b897dc9aa47ff8c3
I20240503 02:12:48.642560 5159 ts_tablet_manager.cc:1909] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.643456 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1098) took 1003 ms. Trace:
I20240503 02:12:48.643560 5159 rpcz_store.cc:276] 0503 02:12:47.639835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.639897 (+ 62us) service_pool.cc:225] Handling call
0503 02:12:48.634377 (+994480us) ts_tablet_manager.cc:1120] Deleting tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3
0503 02:12:48.634390 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.638914 (+ 4524us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.641982 (+ 3068us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.643430 (+ 1448us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":210,"tablet-delete.queue_time_us":994219}
W20240503 02:12:48.644142 4746 consensus_peers.cc:485] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.644264 4746 consensus_peers.cc:485] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.644342 4746 leader_election.cc:334] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.644439 4746 leader_election.cc:302] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.644665 7003 raft_consensus.cc:2726] T bca69a5b48754602b04481fb6a94864f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.645009 4746 leader_election.cc:334] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.645197 4746 leader_election.cc:302] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:48.647225 7003 raft_consensus.cc:2726] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:48.647591 4746 leader_election.cc:334] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
W20240503 02:12:48.647894 4746 leader_election.cc:334] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
W20240503 02:12:48.648214 4746 consensus_peers.cc:485] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.648383 4746 consensus_peers.cc:485] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.648519 4746 consensus_peers.cc:485] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:48.648638 4746 leader_election.cc:334] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
W20240503 02:12:48.648869 4746 leader_election.cc:334] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
W20240503 02:12:48.649137 4746 leader_election.cc:334] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.649276 4746 leader_election.cc:302] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:48.651614 4746 raft_consensus.cc:2670] T 5cd296bb3b574d6cbb0252ad10ada00c P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:48.652247 4746 consensus_peers.cc:485] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.651427 5105 ts_tablet_manager.cc:1367] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.089s user 0.014s sys 0.005s
W20240503 02:12:48.655102 4746 consensus_peers.cc:485] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:48.655721 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 47c7ba3639bd4cb4b897dc9aa47ff8c3 (table default.Table_bb051703036147bd9a71497d00edc307 [id=07ffc27bc99f43a8b690d59638e737f6]) successfully deleted
I20240503 02:12:48.655872 5105 raft_consensus.cc:348] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.656071 5105 raft_consensus.cc:374] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.656132 5105 raft_consensus.cc:729] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.656360 5105 consensus_queue.cc:260] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.667325 7003 raft_consensus.cc:482] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.667515 7003 raft_consensus.cc:504] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.668186 7003 leader_election.cc:288] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:48.671624 4746 leader_election.cc:334] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.672255 5105 ts_tablet_manager.cc:1398] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 02:12:48.674100 7003 raft_consensus.cc:482] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.674233 7003 raft_consensus.cc:504] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.674809 7003 leader_election.cc:288] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:48.681573 4746 leader_election.cc:334] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.692338 5105 tablet_bootstrap.cc:492] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.694242 5105 tablet_bootstrap.cc:654] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.692714 5159 tablet_replica.cc:330] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.695305 5159 raft_consensus.cc:2229] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.695605 5159 raft_consensus.cc:2258] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.696229 5159 ts_tablet_manager.cc:1875] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.703105 5159 ts_tablet_manager.cc:1888] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.703374 5159 log.cc:1196] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/20b6a18be64049049e1a083bf4af51ba
I20240503 02:12:48.703814 5159 ts_tablet_manager.cc:1909] T 20b6a18be64049049e1a083bf4af51ba P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.704746 5159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39380 (request call id 1099) took 1051 ms. Trace:
I20240503 02:12:48.704900 5159 rpcz_store.cc:276] 0503 02:12:47.652982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:47.653050 (+ 68us) service_pool.cc:225] Handling call
0503 02:12:48.692655 (+1039605us) ts_tablet_manager.cc:1120] Deleting tablet 20b6a18be64049049e1a083bf4af51ba
0503 02:12:48.692667 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:48.699530 (+ 6863us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.702979 (+ 3449us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:48.704721 (+ 1742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039354}
I20240503 02:12:48.705471 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 20b6a18be64049049e1a083bf4af51ba (table default.Table_d372fe582377472fb9b1ae98f230a96f [id=03544a9ce3c043bb82e1e86f9fd29609]) successfully deleted
I20240503 02:12:48.705657 5159 tablet_replica.cc:330] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.705897 5159 raft_consensus.cc:2229] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.706143 5159 raft_consensus.cc:2258] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.706748 5159 ts_tablet_manager.cc:1875] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.713582 5159 ts_tablet_manager.cc:1888] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.713862 5159 log.cc:1196] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a0e1c2fa9cb3481dafb9d7f1e826a30c
I20240503 02:12:48.714382 5159 ts_tablet_manager.cc:1909] T a0e1c2fa9cb3481dafb9d7f1e826a30c P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.715754 5159 tablet_replica.cc:330] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.715929 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a0e1c2fa9cb3481dafb9d7f1e826a30c (table default.Table_e8a9dc27e1654f2e92a1bd71999f7959 [id=6d0dbeeacad04c1f8db9512b07d8a0ed]) successfully deleted
I20240503 02:12:48.716149 5159 raft_consensus.cc:2229] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.716378 5105 tablet_bootstrap.cc:492] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.716539 5105 ts_tablet_manager.cc:1367] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:12:48.716540 5159 raft_consensus.cc:2258] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.717394 5105 raft_consensus.cc:348] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.717604 5105 raft_consensus.cc:374] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.717658 5105 raft_consensus.cc:729] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.717856 5105 consensus_queue.cc:260] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.718081 5159 ts_tablet_manager.cc:1875] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.722806 5105 ts_tablet_manager.cc:1398] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:48.729089 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 46005e39ab284da6bf1bb9565318a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.731911 5105 tablet_bootstrap.cc:492] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.734551 6941 raft_consensus.cc:482] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.734740 6941 raft_consensus.cc:504] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.735412 6941 leader_election.cc:288] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.734532 7003 raft_consensus.cc:482] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.735816 7003 raft_consensus.cc:504] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.736434 7003 leader_election.cc:288] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:48.739848 4746 leader_election.cc:334] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
W20240503 02:12:48.740101 4746 leader_election.cc:334] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.769595 5105 tablet_bootstrap.cc:654] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.778599 5159 ts_tablet_manager.cc:1888] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.778841 5159 log.cc:1196] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/905ca3be01e344d38350a58ac9f0d321
I20240503 02:12:48.779285 5159 ts_tablet_manager.cc:1909] T 905ca3be01e344d38350a58ac9f0d321 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.780557 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 905ca3be01e344d38350a58ac9f0d321 (table default.Table_d47e1a80afa040948b01f4ba1d3bf10b [id=74f1b122b8b3437b86a055dbb1222030]) successfully deleted
I20240503 02:12:48.780623 5159 tablet_replica.cc:330] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.780951 5159 raft_consensus.cc:2229] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.781208 5159 raft_consensus.cc:2258] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.781903 5159 ts_tablet_manager.cc:1875] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.789011 5159 ts_tablet_manager.cc:1888] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.789266 5159 log.cc:1196] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/16b3d1e7ce83425db0bd56457d9f2544
I20240503 02:12:48.789710 5159 ts_tablet_manager.cc:1909] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.790575 5159 tablet_replica.cc:330] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.790714 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 16b3d1e7ce83425db0bd56457d9f2544 (table default.Table_9ffe0b87ba144949b561f827d3d06611 [id=5a08204f484f414c9749befba8b8182b]) successfully deleted
I20240503 02:12:48.790918 5159 raft_consensus.cc:2229] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.791141 5159 raft_consensus.cc:2258] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.791769 5159 ts_tablet_manager.cc:1875] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.791786 5105 tablet_bootstrap.cc:492] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.792102 5105 ts_tablet_manager.cc:1367] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.001s
I20240503 02:12:48.792795 5105 raft_consensus.cc:348] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.793032 5105 raft_consensus.cc:374] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.793082 5105 raft_consensus.cc:729] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.793289 5105 consensus_queue.cc:260] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.793797 5105 ts_tablet_manager.cc:1398] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:48.794102 5105 tablet_bootstrap.cc:492] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.795523 5105 tablet_bootstrap.cc:654] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.808588 5159 ts_tablet_manager.cc:1888] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.808862 5159 log.cc:1196] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5b91a3f3cad74f6aa6035a013cca651f
I20240503 02:12:48.809378 5159 ts_tablet_manager.cc:1909] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.809713 6814 raft_consensus.cc:482] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.809878 6814 raft_consensus.cc:504] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.810032 5159 tablet_replica.cc:330] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.810222 5159 raft_consensus.cc:2229] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.810374 5159 raft_consensus.cc:2258] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.810595 6814 leader_election.cc:288] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:48.812152 4746 leader_election.cc:334] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.812793 5159 ts_tablet_manager.cc:1875] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.815634 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e04292742646049f39b6aa29338d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.816174 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 5b91a3f3cad74f6aa6035a013cca651f (table default.Table_7ea27632721644359e27e0917381fb1c [id=5551c6d493d74bc39cea10e5a8198e8a]) successfully deleted
I20240503 02:12:48.817837 6814 raft_consensus.cc:482] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.817987 6814 raft_consensus.cc:504] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.818634 6814 leader_election.cc:288] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.819972 6814 raft_consensus.cc:482] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.820106 6814 raft_consensus.cc:504] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
W20240503 02:12:48.820508 4746 leader_election.cc:334] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.820699 6814 leader_election.cc:288] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.821851 5105 tablet_bootstrap.cc:492] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.821990 5105 ts_tablet_manager.cc:1367] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.005s
W20240503 02:12:48.822139 4746 leader_election.cc:334] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.822688 5105 raft_consensus.cc:348] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.822878 5105 raft_consensus.cc:374] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.822925 5105 raft_consensus.cc:729] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.823132 5105 consensus_queue.cc:260] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.823463 5159 ts_tablet_manager.cc:1888] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.823577 5159 log.cc:1196] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/818788a24cd1419980c0d622fb05683b
I20240503 02:12:48.823653 5105 ts_tablet_manager.cc:1398] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:48.824070 5159 ts_tablet_manager.cc:1909] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.827337 5105 tablet_bootstrap.cc:492] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.828781 5105 tablet_bootstrap.cc:654] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.847668 5105 tablet_bootstrap.cc:492] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.847842 5105 ts_tablet_manager.cc:1367] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20240503 02:12:48.848510 5105 raft_consensus.cc:348] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.848716 5105 raft_consensus.cc:374] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.848773 5105 raft_consensus.cc:729] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.849010 5105 consensus_queue.cc:260] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.849550 5105 ts_tablet_manager.cc:1398] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:48.849860 5105 tablet_bootstrap.cc:492] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.851855 5105 tablet_bootstrap.cc:654] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.864498 5105 tablet_bootstrap.cc:492] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.864670 5105 ts_tablet_manager.cc:1367] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240503 02:12:48.865355 5105 raft_consensus.cc:348] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.865567 5105 raft_consensus.cc:374] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.865621 5105 raft_consensus.cc:729] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.865828 5105 consensus_queue.cc:260] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.866362 5105 ts_tablet_manager.cc:1398] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:48.871423 5105 tablet_bootstrap.cc:492] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.873219 5105 tablet_bootstrap.cc:654] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.884593 5159 tablet_replica.cc:330] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.891896 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 818788a24cd1419980c0d622fb05683b (table default.Table_7e6b24685afa41e388162e59492f00b0 [id=ec36effc4a6d4656b250091b3e482f38]) successfully deleted
I20240503 02:12:48.889237 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 28c1a81589df496b92b980353f00e594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.889856 5105 tablet_bootstrap.cc:492] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.891162 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aef6b021da4f5aad7f523b15d484ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.897109 5105 ts_tablet_manager.cc:1367] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20240503 02:12:48.897461 5159 raft_consensus.cc:2229] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.897858 5105 raft_consensus.cc:348] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.904544 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa3df92b6e544de9ca76d98706eebec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:39 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.906354 5105 raft_consensus.cc:374] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.906459 5105 raft_consensus.cc:729] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.906695 5105 consensus_queue.cc:260] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.907195 5159 raft_consensus.cc:2258] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.907498 5105 ts_tablet_manager.cc:1398] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 02:12:48.907928 5159 ts_tablet_manager.cc:1875] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.908054 5105 tablet_bootstrap.cc:492] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.909757 5105 tablet_bootstrap.cc:654] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.929373 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a082f4c43fd4c188be9ee3966e26edf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:41 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.934085 5105 tablet_bootstrap.cc:492] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.934288 5105 ts_tablet_manager.cc:1367] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.015s
I20240503 02:12:48.935004 7003 raft_consensus.cc:482] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.934981 5105 raft_consensus.cc:348] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.935199 5105 raft_consensus.cc:374] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.935145 7003 raft_consensus.cc:504] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.935245 5105 raft_consensus.cc:729] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.935424 5105 consensus_queue.cc:260] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.935768 7003 leader_election.cc:288] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:48.935923 5105 ts_tablet_manager.cc:1398] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:48.936184 5105 tablet_bootstrap.cc:492] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:48.937582 4746 leader_election.cc:334] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.937757 5105 tablet_bootstrap.cc:654] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.938589 6814 raft_consensus.cc:482] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.938714 6814 raft_consensus.cc:504] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.939287 6814 leader_election.cc:288] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:48.941458 4746 leader_election.cc:334] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:48.951231 5159 ts_tablet_manager.cc:1888] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.951421 5159 log.cc:1196] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/676d5f52102d496081df3964722b286f
I20240503 02:12:48.951774 5159 ts_tablet_manager.cc:1909] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.952486 5159 tablet_replica.cc:330] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.952685 5159 raft_consensus.cc:2229] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.952848 5159 raft_consensus.cc:2258] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.953081 5105 tablet_bootstrap.cc:492] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.953334 5105 ts_tablet_manager.cc:1367] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240503 02:12:48.953398 5159 ts_tablet_manager.cc:1875] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.954027 5105 raft_consensus.cc:348] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.954324 5105 raft_consensus.cc:374] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.954396 5105 raft_consensus.cc:729] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.954645 5105 consensus_queue.cc:260] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.955134 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 857d019b8ab94be1b7719fec83e57e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:33 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.955190 5105 ts_tablet_manager.cc:1398] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:48.955444 5105 tablet_bootstrap.cc:492] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.957186 5105 tablet_bootstrap.cc:654] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.961818 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 676d5f52102d496081df3964722b286f (table default.Table_681b4fa6666c4e9aaddf611003dcaa44 [id=942a3d66711342d6ac832486f109644a]) successfully deleted
I20240503 02:12:48.964313 5159 ts_tablet_manager.cc:1888] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.964475 5159 log.cc:1196] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/a88710ac9062472f80bf6fcfb82793a0
I20240503 02:12:48.964777 5159 ts_tablet_manager.cc:1909] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.965426 5159 tablet_replica.cc:330] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.965524 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a43e428b804729adf9d8a3133c90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.965672 5159 raft_consensus.cc:2229] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.965834 5159 raft_consensus.cc:2258] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.966413 5159 ts_tablet_manager.cc:1875] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.966722 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet a88710ac9062472f80bf6fcfb82793a0 (table default.Table_63229f7088d0478b9de9fa953f84866b [id=5fcbb7a62cd3485cb064c56b07c463cf]) successfully deleted
I20240503 02:12:48.969440 5105 tablet_bootstrap.cc:492] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.969604 5105 ts_tablet_manager.cc:1367] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240503 02:12:48.970206 5105 raft_consensus.cc:348] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.970389 5105 raft_consensus.cc:374] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.970448 5105 raft_consensus.cc:729] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.970669 5105 consensus_queue.cc:260] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.971132 5105 ts_tablet_manager.cc:1398] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:48.971904 5105 tablet_bootstrap.cc:492] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:48.972491 5159 ts_tablet_manager.cc:1888] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:48.972631 5159 log.cc:1196] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fdc2850c07d042a7951a5fbeb0e4945d
I20240503 02:12:48.972899 5159 ts_tablet_manager.cc:1909] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:48.977069 5159 tablet_replica.cc:330] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:48.977320 5159 raft_consensus.cc:2229] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:48.977488 5159 raft_consensus.cc:2258] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:48.978487 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fdc2850c07d042a7951a5fbeb0e4945d (table default.Table_86bc1fe3319346129d38c1c2d731742f [id=974d99380a264ecc962b379edadca637]) successfully deleted
I20240503 02:12:48.978688 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet b9470480a9de45209044f8e8c461519a with delete_type TABLET_DATA_DELETED (replaced by tablet 620223977a5f48209f6cc12b53915508) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:48.979241 5159 ts_tablet_manager.cc:1875] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:48.985831 5105 tablet_bootstrap.cc:654] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.988024 5105 tablet_bootstrap.cc:492] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:48.988272 5105 ts_tablet_manager.cc:1367] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 02:12:48.989876 5105 raft_consensus.cc:348] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.990197 5105 raft_consensus.cc:374] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.990319 5105 raft_consensus.cc:729] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.990608 5105 consensus_queue.cc:260] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:48.996629 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet d6c63f889cb54619927ade6989fed63a (table default.Table_08e73f7a6e1f427b8311fef7e8821883 [id=c0a47dd6f6f54b9b8f752dc5a51ea8c9]). Expected version 1 got 0
I20240503 02:12:48.999655 5105 ts_tablet_manager.cc:1398] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:49.000200 5159 ts_tablet_manager.cc:1888] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.000349 5159 log.cc:1196] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8f8b6599206549dd8ddf449a52c61fce
I20240503 02:12:49.000651 5159 ts_tablet_manager.cc:1909] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.001478 5105 tablet_bootstrap.cc:492] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.003247 5105 tablet_bootstrap.cc:654] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.004534 5159 tablet_replica.cc:330] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.004757 5159 raft_consensus.cc:2229] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.004823 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8f8b6599206549dd8ddf449a52c61fce (table default.Table_453e312025324be6a4d15ae5116a46fa [id=c11dd6cb733044ef94c59992e98601f1]) successfully deleted
I20240503 02:12:49.004949 5159 raft_consensus.cc:2258] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.005779 5159 ts_tablet_manager.cc:1875] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.010596 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet d6c63f889cb54619927ade6989fed63a (table default.Table_08e73f7a6e1f427b8311fef7e8821883 [id=c0a47dd6f6f54b9b8f752dc5a51ea8c9]). Expected version 1 got 0
I20240503 02:12:49.012334 5159 ts_tablet_manager.cc:1888] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.012477 5159 log.cc:1196] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/46005e39ab284da6bf1bb9565318a2f1
I20240503 02:12:49.012779 5159 ts_tablet_manager.cc:1909] T 46005e39ab284da6bf1bb9565318a2f1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.013883 5105 tablet_bootstrap.cc:492] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.014036 5105 ts_tablet_manager.cc:1367] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:12:49.014865 5105 raft_consensus.cc:348] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.015070 5105 raft_consensus.cc:374] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.015120 5105 raft_consensus.cc:729] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.015136 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 46005e39ab284da6bf1bb9565318a2f1 (table default.Table_a2295477032745c3b7b2839c77679705 [id=842fcbdfc3ea408ea71b929b54751160]) successfully deleted
I20240503 02:12:49.015326 5105 consensus_queue.cc:260] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.014983 5159 tablet_replica.cc:330] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.015836 5105 ts_tablet_manager.cc:1398] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:49.016546 5159 raft_consensus.cc:2229] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.016664 5105 tablet_bootstrap.cc:492] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.016702 5159 raft_consensus.cc:2258] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.018433 5105 tablet_bootstrap.cc:654] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.019606 5159 ts_tablet_manager.cc:1875] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.021960 7003 raft_consensus.cc:482] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.022133 7003 raft_consensus.cc:504] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.022799 7003 leader_election.cc:288] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:49.022914 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa77e6c9a494fa2bdc74c5d725624cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.026530 5105 tablet_bootstrap.cc:492] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.026703 5105 ts_tablet_manager.cc:1367] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240503 02:12:49.027526 5105 raft_consensus.cc:348] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.027719 5105 raft_consensus.cc:374] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.027763 5105 raft_consensus.cc:729] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
W20240503 02:12:49.027781 4746 leader_election.cc:334] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.027987 5105 consensus_queue.cc:260] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.028498 5105 ts_tablet_manager.cc:1398] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:49.028760 5105 tablet_bootstrap.cc:492] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.030407 5105 tablet_bootstrap.cc:654] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.033483 5105 tablet_bootstrap.cc:492] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.033643 5105 ts_tablet_manager.cc:1367] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:49.034416 5105 raft_consensus.cc:348] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.034636 5105 raft_consensus.cc:374] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.034693 5105 raft_consensus.cc:729] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.036185 5159 ts_tablet_manager.cc:1888] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.036399 5159 log.cc:1196] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/50e04292742646049f39b6aa29338d75
I20240503 02:12:49.036850 5159 ts_tablet_manager.cc:1909] T 50e04292742646049f39b6aa29338d75 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.037815 5105 consensus_queue.cc:260] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.038005 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 50e04292742646049f39b6aa29338d75 (table default.Table_0134301a980f41e5847132881085032e [id=d0177c0786914f7cb381e9693c930937]) successfully deleted
I20240503 02:12:49.038372 5105 ts_tablet_manager.cc:1398] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:49.038661 5105 tablet_bootstrap.cc:492] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.040393 5105 tablet_bootstrap.cc:654] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.041325 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d555c8d8844ee49c960a5279485480 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:34 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.046773 5159 tablet_replica.cc:330] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.047039 5159 raft_consensus.cc:2229] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.047312 5159 raft_consensus.cc:2258] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.048027 5159 ts_tablet_manager.cc:1875] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.048527 5105 tablet_bootstrap.cc:492] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.048734 5105 ts_tablet_manager.cc:1367] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 02:12:49.049635 5105 raft_consensus.cc:348] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.049928 5105 raft_consensus.cc:374] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.050036 5105 raft_consensus.cc:729] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.050339 5105 consensus_queue.cc:260] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.055092 5159 ts_tablet_manager.cc:1888] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.055255 5159 log.cc:1196] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/28c1a81589df496b92b980353f00e594
I20240503 02:12:49.055651 5159 ts_tablet_manager.cc:1909] T 28c1a81589df496b92b980353f00e594 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.056663 5159 tablet_replica.cc:330] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.057232 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 28c1a81589df496b92b980353f00e594 (table default.Table_6d070a69982c45b7904a12c6df0386b6 [id=6f362fd85655487d93025664e6a68fbe]) successfully deleted
I20240503 02:12:49.057498 5105 ts_tablet_manager.cc:1398] T 3db446d8a6984d9cb4175887f7e125bb P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:49.056874 5159 raft_consensus.cc:2229] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.058915 5159 raft_consensus.cc:2258] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.059382 5105 tablet_bootstrap.cc:492] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.061195 5105 tablet_bootstrap.cc:654] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.064460 5159 ts_tablet_manager.cc:1875] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.066457 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet fca2c34e0fc44be8b7ee9252ef557497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.070917 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 3db446d8a6984d9cb4175887f7e125bb (table default.Table_f24dcaf9ccd649c9ad2afc1764f3e6d5 [id=7a8979a4524b4879880b449477f750ed]). Expected version 1 got 0
I20240503 02:12:49.071763 5159 ts_tablet_manager.cc:1888] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.079218 5105 tablet_bootstrap.cc:492] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.079416 5105 ts_tablet_manager.cc:1367] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.003s
I20240503 02:12:49.080276 5105 raft_consensus.cc:348] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.080513 5105 raft_consensus.cc:374] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.080571 5105 raft_consensus.cc:729] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.080792 5105 consensus_queue.cc:260] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.081370 5105 ts_tablet_manager.cc:1398] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:49.081673 5105 tablet_bootstrap.cc:492] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.083314 5105 tablet_bootstrap.cc:654] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.091475 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a047360ed8843008c4ba633a65b2769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.095717 5159 log.cc:1196] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/76aef6b021da4f5aad7f523b15d484ec
I20240503 02:12:49.096086 5159 ts_tablet_manager.cc:1909] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.096979 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 76aef6b021da4f5aad7f523b15d484ec (table default.Table_4f5be79fd35347a7ac91d7cc508b4a67 [id=b078c65aa39343aabe02f29bf833abe6]) successfully deleted
I20240503 02:12:49.097427 5159 tablet_replica.cc:330] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.097748 5159 raft_consensus.cc:2229] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.097924 5159 raft_consensus.cc:2258] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.098652 5159 ts_tablet_manager.cc:1875] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.102012 5105 tablet_bootstrap.cc:492] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.102193 5105 ts_tablet_manager.cc:1367] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.015s
I20240503 02:12:49.102905 5105 raft_consensus.cc:348] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.103119 5105 raft_consensus.cc:374] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.103166 5105 raft_consensus.cc:729] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.103183 5159 ts_tablet_manager.cc:1888] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.103297 5159 log.cc:1196] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/9fa3df92b6e544de9ca76d98706eebec
I20240503 02:12:49.103564 5159 ts_tablet_manager.cc:1909] T 9fa3df92b6e544de9ca76d98706eebec P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.103442 5105 consensus_queue.cc:260] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.104053 5105 ts_tablet_manager.cc:1398] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:49.104346 5105 tablet_bootstrap.cc:492] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.104665 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 9fa3df92b6e544de9ca76d98706eebec (table default.Table_1710b5cf16aa49c6b52e5ea46c8e30d8 [id=d636c15b30004d098fa523ef493335db]) successfully deleted
I20240503 02:12:49.105777 5105 tablet_bootstrap.cc:654] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.108945 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 054d57b190ef42ba81f1c8a2cf00761b (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]). Expected version 2 got 0
I20240503 02:12:49.109050 5159 tablet_replica.cc:330] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.109249 5159 raft_consensus.cc:2229] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.109454 5159 raft_consensus.cc:2258] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:49.109583 4271 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 054d57b190ef42ba81f1c8a2cf00761b (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 054d57b190ef42ba81f1c8a2cf00761b consensus state has no leader
I20240503 02:12:49.110354 5159 ts_tablet_manager.cc:1875] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.117017 5159 ts_tablet_manager.cc:1888] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.117180 5159 log.cc:1196] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1a082f4c43fd4c188be9ee3966e26edf
I20240503 02:12:49.117519 5159 ts_tablet_manager.cc:1909] T 1a082f4c43fd4c188be9ee3966e26edf P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.118032 5105 tablet_bootstrap.cc:492] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.118261 5105 ts_tablet_manager.cc:1367] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 02:12:49.119031 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1a082f4c43fd4c188be9ee3966e26edf (table default.Table_a567b7af96934be8836963c85cc81473 [id=9c42710fdba04850b53882a01803eeec]) successfully deleted
I20240503 02:12:49.119176 5159 tablet_replica.cc:330] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.119370 5159 raft_consensus.cc:2229] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.119513 5159 raft_consensus.cc:2258] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.120214 5159 ts_tablet_manager.cc:1875] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.120301 5105 raft_consensus.cc:348] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.120494 5105 raft_consensus.cc:374] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.120548 5105 raft_consensus.cc:729] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.120802 5105 consensus_queue.cc:260] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.123664 4271 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 054d57b190ef42ba81f1c8a2cf00761b (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0]). Expected version 2 got 0
I20240503 02:12:49.126901 5159 ts_tablet_manager.cc:1888] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.127058 5159 log.cc:1196] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/857d019b8ab94be1b7719fec83e57e55
I20240503 02:12:49.127389 5159 ts_tablet_manager.cc:1909] T 857d019b8ab94be1b7719fec83e57e55 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.128451 5159 tablet_replica.cc:330] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.128676 5159 raft_consensus.cc:2229] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.128844 5159 raft_consensus.cc:2258] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.129806 5159 ts_tablet_manager.cc:1875] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.131027 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 857d019b8ab94be1b7719fec83e57e55 (table default.Table_c713f241c06a4a7c8106ba83bcab0c70 [id=ebbf977116684eaf914cb6c4807e73c4]) successfully deleted
I20240503 02:12:49.131215 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecfd29eddde4c39b7661b1b7d96a08e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:25 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.132871 5105 ts_tablet_manager.cc:1398] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 02:12:49.135818 7003 raft_consensus.cc:482] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:49.135982 7003 raft_consensus.cc:504] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.136552 7003 leader_election.cc:288] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:49.138706 4746 leader_election.cc:334] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.138794 5105 tablet_bootstrap.cc:492] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.140642 5159 ts_tablet_manager.cc:1888] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.140777 5159 log.cc:1196] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/50a43e428b804729adf9d8a3133c90ff
I20240503 02:12:49.141098 5159 ts_tablet_manager.cc:1909] T 50a43e428b804729adf9d8a3133c90ff P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.142066 5159 tablet_replica.cc:330] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.142268 5159 raft_consensus.cc:2229] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.142298 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 50a43e428b804729adf9d8a3133c90ff (table default.Table_01a2313a2c52424386a931aeecfac97c [id=7aab2fb7ae69485da4ccef5840ae3aa2]) successfully deleted
I20240503 02:12:49.142428 5159 raft_consensus.cc:2258] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.142777 5105 tablet_bootstrap.cc:654] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.143028 5159 ts_tablet_manager.cc:1875] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.147328 5105 tablet_bootstrap.cc:492] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.147495 5105 ts_tablet_manager.cc:1367] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:12:49.148169 5105 raft_consensus.cc:348] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.148376 5105 raft_consensus.cc:374] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.148435 5105 raft_consensus.cc:729] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.148644 5105 consensus_queue.cc:260] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.151727 5105 ts_tablet_manager.cc:1398] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:49.153537 5159 ts_tablet_manager.cc:1888] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.153743 5159 log.cc:1196] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/b9470480a9de45209044f8e8c461519a
I20240503 02:12:49.154047 5159 ts_tablet_manager.cc:1909] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.154711 5159 tablet_replica.cc:330] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.154985 5159 raft_consensus.cc:2229] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.155157 5159 raft_consensus.cc:2258] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.156805 5105 tablet_bootstrap.cc:492] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.158385 5105 tablet_bootstrap.cc:654] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.160356 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 8241a88f5c3a47578c92343f9fb5f7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.160367 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet b9470480a9de45209044f8e8c461519a (table default.Table_c5703a4668b04accbd9b3f93427b31ab [id=838900269aa04721a4033aab3345c5dc]) successfully deleted
I20240503 02:12:49.161648 5159 ts_tablet_manager.cc:1875] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.163550 5105 tablet_bootstrap.cc:492] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.163705 5105 ts_tablet_manager.cc:1367] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:12:49.164239 5105 raft_consensus.cc:348] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.164391 5105 raft_consensus.cc:374] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.164425 5105 raft_consensus.cc:729] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.164609 5105 consensus_queue.cc:260] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.165130 5105 ts_tablet_manager.cc:1398] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:12:49.167517 5159 ts_tablet_manager.cc:1888] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.167655 5159 log.cc:1196] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/aaa77e6c9a494fa2bdc74c5d725624cb
I20240503 02:12:49.167959 5159 ts_tablet_manager.cc:1909] T aaa77e6c9a494fa2bdc74c5d725624cb P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.168951 5159 tablet_replica.cc:330] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.169183 5159 raft_consensus.cc:2229] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.169344 5159 raft_consensus.cc:2258] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.169983 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet aaa77e6c9a494fa2bdc74c5d725624cb (table default.Table_93262e3fdea24977a7a3aa557373b9eb [id=501dab2edaf84e3aaceab4bd5984617c]) successfully deleted
I20240503 02:12:49.170454 5105 tablet_bootstrap.cc:492] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.172194 5105 tablet_bootstrap.cc:654] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.177726 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b92999890ee49d78d7c16f5ed1c4958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:40 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.178229 5159 ts_tablet_manager.cc:1875] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.184163 5105 tablet_bootstrap.cc:492] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.184329 5105 ts_tablet_manager.cc:1367] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20240503 02:12:49.185032 5105 raft_consensus.cc:348] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.185209 5105 raft_consensus.cc:374] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.185266 5105 raft_consensus.cc:729] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.186333 5159 ts_tablet_manager.cc:1888] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.186467 5159 log.cc:1196] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/04d555c8d8844ee49c960a5279485480
I20240503 02:12:49.186755 5159 ts_tablet_manager.cc:1909] T 04d555c8d8844ee49c960a5279485480 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.185479 5105 consensus_queue.cc:260] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.187676 5105 ts_tablet_manager.cc:1398] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:49.190627 5105 tablet_bootstrap.cc:492] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.192260 5105 tablet_bootstrap.cc:654] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.193306 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bfcc73e2eb57429aa61b52b42198c747 (table default.Table_6c31c5686db9455c836948d8e8abbfe8 [id=7230242df5244c83974d492b97d838a7]). Expected version 2 got 0
I20240503 02:12:49.194975 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 04d555c8d8844ee49c960a5279485480 (table default.Table_f471b5f3094048268179ff4ff91b119f [id=f95b06fca203484c90903ffc1a2dd58f]) successfully deleted
I20240503 02:12:49.196775 5159 tablet_replica.cc:330] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.197021 5159 raft_consensus.cc:2229] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.197187 5159 raft_consensus.cc:2258] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.201740 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet bfcc73e2eb57429aa61b52b42198c747 (table default.Table_6c31c5686db9455c836948d8e8abbfe8 [id=7230242df5244c83974d492b97d838a7]). Expected version 2 got 0
I20240503 02:12:49.203436 7003 raft_consensus.cc:482] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.203617 7003 raft_consensus.cc:504] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.204038 5159 ts_tablet_manager.cc:1875] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:49.204108 4260 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 054d57b190ef42ba81f1c8a2cf00761b (table default.Table_8e1cb6d17d9242f9903e21c37cbb3f72 [id=0689a370309d457490de9ead26d200f0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 054d57b190ef42ba81f1c8a2cf00761b consensus state has no leader
I20240503 02:12:49.204246 7003 leader_election.cc:288] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:49.205976 4746 leader_election.cc:334] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.207701 5105 tablet_bootstrap.cc:492] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.207849 5105 ts_tablet_manager.cc:1367] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240503 02:12:49.208518 5105 raft_consensus.cc:348] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.208710 5105 raft_consensus.cc:374] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.208755 5105 raft_consensus.cc:729] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.208977 5105 consensus_queue.cc:260] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.209492 5105 ts_tablet_manager.cc:1398] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.210273 5159 ts_tablet_manager.cc:1888] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.210366 5159 log.cc:1196] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fca2c34e0fc44be8b7ee9252ef557497
I20240503 02:12:49.210665 5159 ts_tablet_manager.cc:1909] T fca2c34e0fc44be8b7ee9252ef557497 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.215958 5159 tablet_replica.cc:330] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.216181 5159 raft_consensus.cc:2229] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.216326 5159 raft_consensus.cc:2258] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.217056 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 368aabce19984d03aa7116c8df990ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:45 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.217367 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet fca2c34e0fc44be8b7ee9252ef557497 (table default.Table_bfb8401dc518472abd4e26a2c788377d [id=10893601d3fa4c87b63e5c8acc68bce5]) successfully deleted
I20240503 02:12:49.222674 5159 ts_tablet_manager.cc:1875] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.223304 5105 tablet_bootstrap.cc:492] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.225174 5105 tablet_bootstrap.cc:654] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:49.229406 4746 consensus_peers.cc:485] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 -> Peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Couldn't send request to peer f4f9f079230a468f90e5e3a58f254b91. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:49.229954 5159 ts_tablet_manager.cc:1888] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.230082 5159 log.cc:1196] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/7a047360ed8843008c4ba633a65b2769
I20240503 02:12:49.230373 5159 ts_tablet_manager.cc:1909] T 7a047360ed8843008c4ba633a65b2769 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.231422 5159 tablet_replica.cc:330] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.231637 5159 raft_consensus.cc:2229] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.231787 5159 raft_consensus.cc:2258] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.232597 5159 ts_tablet_manager.cc:1875] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.234164 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 7a047360ed8843008c4ba633a65b2769 (table default.Table_5b3b2e6e211c47338aa5ddca26669a20 [id=f2bff3fc937040f7989f4c1e43c5773a]) successfully deleted
I20240503 02:12:49.237920 5105 tablet_bootstrap.cc:492] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.238068 5105 ts_tablet_manager.cc:1367] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240503 02:12:49.238950 5105 raft_consensus.cc:348] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.239156 5105 raft_consensus.cc:374] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.239212 5105 raft_consensus.cc:729] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.239370 5159 ts_tablet_manager.cc:1888] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.239498 5159 log.cc:1196] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2ecfd29eddde4c39b7661b1b7d96a08e
I20240503 02:12:49.239486 5105 consensus_queue.cc:260] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.239888 5159 ts_tablet_manager.cc:1909] T 2ecfd29eddde4c39b7661b1b7d96a08e P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.240043 5105 ts_tablet_manager.cc:1398] T dc16522e513c44129f4bbad7c31edb9a P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:49.249806 5105 tablet_bootstrap.cc:492] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.251538 5105 tablet_bootstrap.cc:654] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.251778 5159 tablet_replica.cc:330] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.252887 5159 raft_consensus.cc:2229] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.253186 5159 raft_consensus.cc:2258] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.256090 5159 ts_tablet_manager.cc:1875] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.256191 5105 tablet_bootstrap.cc:492] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.256361 5105 ts_tablet_manager.cc:1367] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 02:12:49.257331 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 2ecfd29eddde4c39b7661b1b7d96a08e (table default.Table_df799d52e6434e0f90b831960d0f39c1 [id=19ac7478bcac48f191daaf64dcecdec9]) successfully deleted
I20240503 02:12:49.257282 5105 raft_consensus.cc:348] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.257495 5105 raft_consensus.cc:374] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.257556 5105 raft_consensus.cc:729] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.257843 5105 consensus_queue.cc:260] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.258368 5105 ts_tablet_manager.cc:1398] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:49.263103 5159 ts_tablet_manager.cc:1888] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.266703 5159 log.cc:1196] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/8241a88f5c3a47578c92343f9fb5f7cd
I20240503 02:12:49.267225 5159 ts_tablet_manager.cc:1909] T 8241a88f5c3a47578c92343f9fb5f7cd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.267558 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 620204f7cb35442f87eae45f84beeee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.268875 5159 tablet_replica.cc:330] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.269177 5159 raft_consensus.cc:2229] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.269297 5105 tablet_bootstrap.cc:492] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.269431 5159 raft_consensus.cc:2258] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.270929 5105 tablet_bootstrap.cc:654] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.275740 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 8241a88f5c3a47578c92343f9fb5f7cd (table default.Table_ab30b57f7d9044fca1bf76dc5e2bc323 [id=b7bcb6c138184fdab9ffe2e84b76772c]) successfully deleted
I20240503 02:12:49.277696 5105 tablet_bootstrap.cc:492] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.277647 5159 ts_tablet_manager.cc:1875] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.277860 5105 ts_tablet_manager.cc:1367] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20240503 02:12:49.278658 5105 raft_consensus.cc:348] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.278849 5105 raft_consensus.cc:374] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.278895 5105 raft_consensus.cc:729] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.278921 7003 raft_consensus.cc:482] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.279052 7003 raft_consensus.cc:504] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.279112 5105 consensus_queue.cc:260] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.279574 5105 ts_tablet_manager.cc:1398] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:49.279668 7003 leader_election.cc:288] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:49.284183 5159 ts_tablet_manager.cc:1888] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.284313 5159 log.cc:1196] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/3b92999890ee49d78d7c16f5ed1c4958
I20240503 02:12:49.284617 5159 ts_tablet_manager.cc:1909] T 3b92999890ee49d78d7c16f5ed1c4958 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.284713 5105 tablet_bootstrap.cc:492] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.285531 5159 tablet_replica.cc:330] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.285744 5159 raft_consensus.cc:2229] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.285892 5159 raft_consensus.cc:2258] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.286543 5105 tablet_bootstrap.cc:654] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.288434 5105 tablet_bootstrap.cc:492] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.288584 5105 ts_tablet_manager.cc:1367] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:49.290086 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 3b92999890ee49d78d7c16f5ed1c4958 (table default.Table_de40cf71b0974c799cf87de718e159b4 [id=a8b87d1e66ad42798ddfa4de05f5dbea]) successfully deleted
I20240503 02:12:49.292102 5159 ts_tablet_manager.cc:1875] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:12:49.292675 4746 leader_election.cc:334] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.289371 5105 raft_consensus.cc:348] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.295588 5105 raft_consensus.cc:374] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.295679 5105 raft_consensus.cc:729] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.295933 5105 consensus_queue.cc:260] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.301671 5159 ts_tablet_manager.cc:1888] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.301847 5105 ts_tablet_manager.cc:1398] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 02:12:49.301898 5159 log.cc:1196] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/368aabce19984d03aa7116c8df990ef1
I20240503 02:12:49.302366 5159 ts_tablet_manager.cc:1909] T 368aabce19984d03aa7116c8df990ef1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.306869 5159 tablet_replica.cc:330] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.307111 5159 raft_consensus.cc:2229] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.307271 5159 raft_consensus.cc:2258] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.307658 5105 tablet_bootstrap.cc:492] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.309026 5105 tablet_bootstrap.cc:654] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.313000 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 368aabce19984d03aa7116c8df990ef1 (table default.Table_5e542f394cce413f99f8012fcc9fba9a [id=aef6e606e6f34ed78039dd362329ce77]) successfully deleted
I20240503 02:12:49.313374 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a4436ab9e7542f5b3aca930f6c5a758) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.318171 5159 ts_tablet_manager.cc:1875] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.324981 5159 ts_tablet_manager.cc:1888] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.325129 5159 log.cc:1196] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/620204f7cb35442f87eae45f84beeee1
I20240503 02:12:49.325493 5159 ts_tablet_manager.cc:1909] T 620204f7cb35442f87eae45f84beeee1 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.326836 5159 tablet_replica.cc:330] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.326756 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 620204f7cb35442f87eae45f84beeee1 (table default.Table_b8df46519f1a401b862bbfabff3577fe [id=27c7348bc7034ab8b9a4f483134459a6]) successfully deleted
I20240503 02:12:49.327044 5159 raft_consensus.cc:2229] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.327256 5159 raft_consensus.cc:2258] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.328070 5159 ts_tablet_manager.cc:1875] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.334602 5159 ts_tablet_manager.cc:1888] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.334749 5159 log.cc:1196] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/1b13ea83ffa54dcbbfebaaa76e1686a9
I20240503 02:12:49.335086 5159 ts_tablet_manager.cc:1909] T 1b13ea83ffa54dcbbfebaaa76e1686a9 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.336220 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 1b13ea83ffa54dcbbfebaaa76e1686a9 (table default.Table_731eebc7b98e4b78832e9d33d32e0b5e [id=779c0d9ea5a6420d8ede93243a64261e]) successfully deleted
I20240503 02:12:49.341104 6814 raft_consensus.cc:482] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.341290 6814 raft_consensus.cc:504] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.341606 7082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df63329dc2b149d9b08cea341b540d1d)
I20240503 02:12:49.342021 6814 leader_election.cc:288] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:49.342800 7082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c621d795c44d0daa534261920d874a" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 43321 } } peers { permanent_uuid: "df63329dc2b149d9b08cea341b540d1d" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 46221 } } peers { permanent_uuid: "145afaa7548f4b7fafb23f20276f0cb4" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 46689 } }
W20240503 02:12:49.343865 4746 leader_election.cc:334] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.344189 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1292 } } ignore_live_leader: false dest_uuid: "00c621d795c44d0daa534261920d874a" is_pre_election: true
I20240503 02:12:49.345736 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "145afaa7548f4b7fafb23f20276f0cb4" candidate_term: 6 candidate_status { last_received { term: 5 index: 1292 } } ignore_live_leader: false dest_uuid: "df63329dc2b149d9b08cea341b540d1d" is_pre_election: true
I20240503 02:12:49.346652 6193 leader_election.cc:302] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 145afaa7548f4b7fafb23f20276f0cb4; no voters: 00c621d795c44d0daa534261920d874a, df63329dc2b149d9b08cea341b540d1d
I20240503 02:12:49.350056 5105 tablet_bootstrap.cc:492] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.350235 5105 ts_tablet_manager.cc:1367] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.003s
I20240503 02:12:49.352587 7082 leader_election.cc:288] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 00c621d795c44d0daa534261920d874a (127.4.17.126:43321), df63329dc2b149d9b08cea341b540d1d (127.4.17.125:46221)
I20240503 02:12:49.353735 5105 raft_consensus.cc:348] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.353914 5105 raft_consensus.cc:374] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.353974 5105 raft_consensus.cc:729] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.354195 5105 consensus_queue.cc:260] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.354668 5105 ts_tablet_manager.cc:1398] T cbd9c89388354e87a56a2c38d791bff8 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:49.354919 5105 tablet_bootstrap.cc:492] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.356547 5105 tablet_bootstrap.cc:654] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.353313 7084 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 145afaa7548f4b7fafb23f20276f0cb4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:12:49.365140 6814 raft_consensus.cc:482] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:49.365377 6814 raft_consensus.cc:504] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.366370 6814 leader_election.cc:288] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:49.373469 4746 leader_election.cc:334] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.405340 5105 tablet_bootstrap.cc:492] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.405517 5105 ts_tablet_manager.cc:1367] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.051s user 0.026s sys 0.021s
I20240503 02:12:49.406236 5105 raft_consensus.cc:348] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.406450 5105 raft_consensus.cc:374] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.406498 5105 raft_consensus.cc:729] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.406714 5105 consensus_queue.cc:260] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.407223 5105 ts_tablet_manager.cc:1398] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:49.407557 5105 tablet_bootstrap.cc:492] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.409299 5105 tablet_bootstrap.cc:654] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.417446 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0745a55223475893edd01d904a223b with delete_type TABLET_DATA_DELETED (replaced by tablet 3c421694ce1e424da7953d061a6bca3e) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.417919 5159 tablet_replica.cc:330] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.418162 5159 raft_consensus.cc:2229] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.418337 5159 raft_consensus.cc:2258] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.419189 5159 ts_tablet_manager.cc:1875] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.423312 5105 tablet_bootstrap.cc:492] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.423492 5105 ts_tablet_manager.cc:1367] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240503 02:12:49.424242 5105 raft_consensus.cc:348] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.424465 5105 raft_consensus.cc:374] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.424520 5105 raft_consensus.cc:729] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.424798 5105 consensus_queue.cc:260] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.425218 5159 ts_tablet_manager.cc:1888] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.425324 5105 ts_tablet_manager.cc:1398] T 43fc277fa4884c7f92118e333226e800 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.425612 5105 tablet_bootstrap.cc:492] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.427371 5105 tablet_bootstrap.cc:654] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.425324 5159 log.cc:1196] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/4c0745a55223475893edd01d904a223b
I20240503 02:12:49.431111 5159 ts_tablet_manager.cc:1909] T 4c0745a55223475893edd01d904a223b P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.435659 6941 raft_consensus.cc:482] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.435827 6941 raft_consensus.cc:504] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.435966 5105 tablet_bootstrap.cc:492] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.436111 5105 ts_tablet_manager.cc:1367] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:12:49.436499 6941 leader_election.cc:288] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:49.437342 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 4c0745a55223475893edd01d904a223b (table default.Table_334ae1c85436463dad80d97f857112a0 [id=731ef763ea6048488518162f5f8a7335]) successfully deleted
I20240503 02:12:49.437347 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 43fc277fa4884c7f92118e333226e800 (table default.Table_2d181d60cede45628a69e42c770848f9 [id=c3fac625561b486c9052e6cf111c2838]). Expected version 2 got 0
I20240503 02:12:49.438253 5105 raft_consensus.cc:348] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.438493 5105 raft_consensus.cc:374] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.438550 5105 raft_consensus.cc:729] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.438758 5105 consensus_queue.cc:260] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.439268 5105 ts_tablet_manager.cc:1398] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 02:12:49.439549 5105 tablet_bootstrap.cc:492] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.441216 5105 tablet_bootstrap.cc:654] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:49.441159 4746 leader_election.cc:334] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.453470 6948 raft_consensus.cc:482] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.449611 6899 raft_consensus.cc:482] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.454026 6899 raft_consensus.cc:504] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.454948 6899 leader_election.cc:288] T 038f11066dc34ba9a833f84a54ff20bf P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
I20240503 02:12:49.453707 6948 raft_consensus.cc:504] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.459470 5105 tablet_bootstrap.cc:492] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.459712 5105 ts_tablet_manager.cc:1367] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240503 02:12:49.460662 5105 raft_consensus.cc:348] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.460943 5105 raft_consensus.cc:374] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.461035 5105 raft_consensus.cc:729] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.461329 5105 consensus_queue.cc:260] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.462019 5105 ts_tablet_manager.cc:1398] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.462414 6941 raft_consensus.cc:482] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:49.462556 6941 raft_consensus.cc:504] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.463274 6941 leader_election.cc:288] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
W20240503 02:12:49.463842 5106 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.003s sys 0.003s
I20240503 02:12:49.464989 5105 tablet_bootstrap.cc:492] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405: Bootstrap starting.
W20240503 02:12:49.465178 4746 leader_election.cc:334] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.466823 5105 tablet_bootstrap.cc:654] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:49.469758 4608 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.000s sys 0.020s
W20240503 02:12:49.469933 4608 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.904s user 0.000s sys 0.020s
W20240503 02:12:49.470063 4608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:39837, user_credentials={real_user=slave}} blocked reactor thread for 904670us
I20240503 02:12:49.472996 6948 leader_election.cc:288] T a86ac17982324847949b8a9245329489 P 1f50719d94564844b15e1a9211d48a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), acc6d33969e74c209310005ae911c405 (127.4.17.67:35993)
W20240503 02:12:49.473294 5106 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.899s user 0.003s sys 0.003s
I20240503 02:12:49.474303 5106 tablet_bootstrap.cc:492] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81: No bootstrap required, opened a new log
I20240503 02:12:49.474496 5106 ts_tablet_manager.cc:1367] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81: Time spent bootstrapping tablet: real 0.913s user 0.004s sys 0.004s
I20240503 02:12:49.475350 5106 raft_consensus.cc:348] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.475566 5106 raft_consensus.cc:374] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.475647 5106 raft_consensus.cc:729] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50719d94564844b15e1a9211d48a81, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.475863 5106 consensus_queue.cc:260] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.476466 5106 ts_tablet_manager.cc:1398] T c3710b653aa24b7ca389aa95d7cdd2fe P 1f50719d94564844b15e1a9211d48a81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:49.476773 5106 tablet_bootstrap.cc:492] T 47c7ba3639bd4cb4b897dc9aa47ff8c3 P 1f50719d94564844b15e1a9211d48a81: Bootstrap starting.
W20240503 02:12:49.477051 4715 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:57760 (request call id 152) took 909 ms (client timeout 1000 ms). Trace:
W20240503 02:12:49.477192 4715 rpcz_store.cc:269] 0503 02:12:48.568041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:48.568105 (+ 64us) service_pool.cc:225] Handling call
0503 02:12:49.477022 (+908917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:12:49.479699 6897 raft_consensus.cc:482] T 20b6a18be64049049e1a083bf4af51ba P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.479789 7021 raft_consensus.cc:482] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.479897 7021 raft_consensus.cc:504] T 5cd296bb3b574d6cbb0252ad10ada00c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.480144 6899 raft_consensus.cc:482] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.480284 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4585
I20240503 02:12:49.480276 6899 raft_consensus.cc:504] T a0e1c2fa9cb3481dafb9d7f1e826a30c P 1f50719d94564844b15e1a9211d48a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.481338 6941 raft_consensus.cc:482] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.481496 6941 raft_consensus.cc:504] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.482122 6941 leader_election.cc:288] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
W20240503 02:12:49.484269 4746 leader_election.cc:334] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.489923 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ccd25074db40fb96053b915125a425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:26 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.490288 5159 tablet_replica.cc:330] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.490592 5159 raft_consensus.cc:2229] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.490798 5159 raft_consensus.cc:2258] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.491714 5159 ts_tablet_manager.cc:1875] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.497313 6941 raft_consensus.cc:482] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.497505 6941 raft_consensus.cc:504] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.498303 6941 leader_election.cc:288] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:49.499548 5105 tablet_bootstrap.cc:492] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.499806 5105 ts_tablet_manager.cc:1367] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.008s
W20240503 02:12:49.500166 4746 leader_election.cc:334] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837): Network error: Client connection negotiation failed: client connection to 127.4.17.65:39837: connect: Connection refused (error 111)
I20240503 02:12:49.500903 5105 raft_consensus.cc:348] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.501255 5105 raft_consensus.cc:374] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.501368 5105 raft_consensus.cc:729] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.501389 5159 ts_tablet_manager.cc:1888] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.501648 5159 log.cc:1196] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/23ccd25074db40fb96053b915125a425
I20240503 02:12:49.501878 5105 consensus_queue.cc:260] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.502089 5159 ts_tablet_manager.cc:1909] T 23ccd25074db40fb96053b915125a425 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.503295 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 23ccd25074db40fb96053b915125a425 (table default.Table_8eb9b75d260a42229a49d6343e0b2e3a [id=0345d41c213e43ef875c7d3e898b3fdb]) successfully deleted
I20240503 02:12:49.505338 5105 ts_tablet_manager.cc:1398] T 6f7ca2fea26749cfa4e4213792c778ce P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:49.506484 5105 tablet_bootstrap.cc:492] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.508533 5105 tablet_bootstrap.cc:654] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.514750 5105 tablet_bootstrap.cc:492] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.515026 5105 ts_tablet_manager.cc:1367] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:49.516198 5105 raft_consensus.cc:348] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.516518 5105 raft_consensus.cc:374] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.516623 5105 raft_consensus.cc:729] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.516897 5105 consensus_queue.cc:260] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.517680 5105 ts_tablet_manager.cc:1398] T 1fa25fee059b46ec915bcc1a01cb8b27 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:49.518029 5105 tablet_bootstrap.cc:492] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.520231 5105 tablet_bootstrap.cc:654] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.528649 5105 tablet_bootstrap.cc:492] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.528996 5105 ts_tablet_manager.cc:1367] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240503 02:12:49.530043 5105 raft_consensus.cc:348] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.530354 5105 raft_consensus.cc:374] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.530467 5105 raft_consensus.cc:729] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.530776 5105 consensus_queue.cc:260] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.531483 5105 ts_tablet_manager.cc:1398] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.531699 4268 catalog_manager.cc:5576] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993) does not have the latest schema for tablet 1fa25fee059b46ec915bcc1a01cb8b27 (table default.Table_1bda6b4dcbff43fc9555244a5fa395f8 [id=e68bbe8a4fd14c1a86f5b81cf8bc06e3]). Expected version 2 got 0
I20240503 02:12:49.531843 5105 tablet_bootstrap.cc:492] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.534044 5105 tablet_bootstrap.cc:654] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.542984 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af50eb2b1c949a0a6a71e64a894e8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.543311 5159 tablet_replica.cc:330] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.543532 5159 raft_consensus.cc:2229] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.543711 5159 raft_consensus.cc:2258] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.544752 5159 ts_tablet_manager.cc:1875] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.551851 5159 ts_tablet_manager.cc:1888] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.552086 5159 log.cc:1196] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/0af50eb2b1c949a0a6a71e64a894e8a5
I20240503 02:12:49.552493 5159 ts_tablet_manager.cc:1909] T 0af50eb2b1c949a0a6a71e64a894e8a5 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.553782 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 0af50eb2b1c949a0a6a71e64a894e8a5 (table default.Table_301d83f516e84342a226f385300cef79 [id=8d41860986c04f2b9b365c0af0744299]) successfully deleted
I20240503 02:12:49.558472 5105 tablet_bootstrap.cc:492] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.558655 5105 ts_tablet_manager.cc:1367] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.004s
I20240503 02:12:49.559628 5105 raft_consensus.cc:348] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.559883 5105 raft_consensus.cc:374] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.559952 5105 raft_consensus.cc:729] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.560195 5105 consensus_queue.cc:260] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.560806 5105 ts_tablet_manager.cc:1398] T 38614de3642148aa82e5108d9160f537 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:49.564005 5105 tablet_bootstrap.cc:492] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.565827 5105 tablet_bootstrap.cc:654] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.577236 5105 tablet_bootstrap.cc:492] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.577491 5105 ts_tablet_manager.cc:1367] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:12:49.578475 5105 raft_consensus.cc:348] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.578717 5105 raft_consensus.cc:374] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.578784 5105 raft_consensus.cc:729] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.579042 5105 consensus_queue.cc:260] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:49.579751 5105 ts_tablet_manager.cc:1398] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240503 02:12:49.582907 5402 connection.cc:535] server connection from 127.4.17.66:35851 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:49.583055 6195 connection.cc:535] server connection from 127.4.17.66:56341 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:49.583411 4259 connection.cc:535] client connection to 127.4.17.66:35355 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:49.584096 4749 connection.cc:535] server connection from 127.4.17.66:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:49.585119 5105 tablet_bootstrap.cc:492] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.585978 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4730
W20240503 02:12:49.586844 4746 connection.cc:535] client connection to 127.4.17.66:35355 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:49.587334 5105 tablet_bootstrap.cc:654] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:49.587621 4259 connection.cc:535] server connection from 127.4.17.66:56851 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:49.603082 4746 consensus_peers.cc:485] T bd0d62ff4ac2460a9620a8e16e5d6548 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.603250 4746 leader_election.cc:334] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.603348 4746 leader_election.cc:302] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.603528 4746 raft_consensus.cc:2670] T b9470480a9de45209044f8e8c461519a P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.604007 4746 consensus_peers.cc:485] T 2c5d58f97b58481ea1dfa75397a32ae2 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.604102 4746 consensus_peers.cc:485] T c2efdbfa909c475d95e99cf24206608f P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.604238 4746 leader_election.cc:334] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.604313 4746 leader_election.cc:302] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.604506 4746 raft_consensus.cc:2670] T fdc2850c07d042a7951a5fbeb0e4945d P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.604954 4746 leader_election.cc:334] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.605099 4746 leader_election.cc:302] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.605378 4746 raft_consensus.cc:2670] T 76aef6b021da4f5aad7f523b15d484ec P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.606015 4746 consensus_peers.cc:485] T b32c377006a24372859ed44d59cf90ee P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.606154 4746 leader_election.cc:334] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.606308 4746 leader_election.cc:302] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.606601 6941 raft_consensus.cc:2726] T ed4e84a639b2403082b737d073f00ad7 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.607062 4746 leader_election.cc:334] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.607304 4746 leader_election.cc:302] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.607563 4746 raft_consensus.cc:2670] T 8f8b6599206549dd8ddf449a52c61fce P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.608243 4746 consensus_peers.cc:485] T a73ec6543b9a468492f3b97772ce43f9 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:49.608824 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a24b5c83b649e79644b08253d851d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:32 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.609146 5159 tablet_replica.cc:330] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.609459 5159 raft_consensus.cc:2229] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.609639 5159 raft_consensus.cc:2258] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:49.609133 4746 leader_election.cc:334] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.610054 4746 leader_election.cc:302] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.610333 4746 raft_consensus.cc:2670] T 676d5f52102d496081df3964722b286f P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.611044 4746 leader_election.cc:334] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.611814 5159 ts_tablet_manager.cc:1875] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.613852 5105 tablet_bootstrap.cc:492] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.614071 5105 ts_tablet_manager.cc:1367] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.004s
I20240503 02:12:49.615068 5105 raft_consensus.cc:348] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.615368 5105 raft_consensus.cc:374] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.615466 5105 raft_consensus.cc:729] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.615787 5105 consensus_queue.cc:260] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.616394 5105 ts_tablet_manager.cc:1398] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 02:12:49.616609 4259 catalog_manager.cc:4573] TS 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d607da9689c14e51add9178a48b35913: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.617036 5105 tablet_bootstrap.cc:492] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.618960 5105 tablet_bootstrap.cc:654] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.625409 4746 leader_election.cc:302] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.625905 4746 raft_consensus.cc:2670] T a88710ac9062472f80bf6fcfb82793a0 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.626576 4746 leader_election.cc:334] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.633080 4746 leader_election.cc:302] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.633503 4746 raft_consensus.cc:2670] T 818788a24cd1419980c0d622fb05683b P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 02:12:49.634248 4746 consensus_peers.cc:485] T d19ed19da5b74583bfc3b21d88e937c9 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.634498 4746 leader_election.cc:334] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.634661 4746 leader_election.cc:302] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.635016 6941 raft_consensus.cc:2726] T 321efd02c5204013b253b43cacdd2e92 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.635489 4746 consensus_peers.cc:485] T c63499fb112c48238d6b84110e394779 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.635679 4746 leader_election.cc:334] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.635829 4746 leader_election.cc:302] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.636121 6941 raft_consensus.cc:2726] T e40cb35a8d544e85ae13377724ed4727 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:12:49.636477 4746 leader_election.cc:334] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.636654 4746 leader_election.cc:302] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.636907 6941 raft_consensus.cc:2726] T 9df090ed6179485eb6922240e4c11ad1 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.637267 4746 leader_election.cc:334] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.637455 4746 leader_election.cc:302] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.637753 6941 raft_consensus.cc:2726] T 89b57a7235b546769c30745dd4c3b9f5 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.638072 4746 leader_election.cc:334] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.638252 4746 leader_election.cc:302] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.638588 6941 raft_consensus.cc:2726] T e366ce9d79b3458fb26e96b6e9278fbe P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:12:49.638960 4746 leader_election.cc:334] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.639138 4746 leader_election.cc:302] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.639498 6941 raft_consensus.cc:2726] T d6c63f889cb54619927ade6989fed63a P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.639904 4746 leader_election.cc:334] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.640087 4746 leader_election.cc:302] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.640383 6941 raft_consensus.cc:2726] T bfcc73e2eb57429aa61b52b42198c747 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.640646 4746 consensus_peers.cc:485] T b743e99847bb47d1ae437c96789a8ea7 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.640826 4746 consensus_peers.cc:485] T 7d919b9262ac48cbb8b3f893918b5590 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.641018 4746 leader_election.cc:334] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.641155 4746 leader_election.cc:302] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.641822 6941 raft_consensus.cc:2726] T 054d57b190ef42ba81f1c8a2cf00761b P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:49.642235 4746 leader_election.cc:334] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.642432 4746 leader_election.cc:302] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
I20240503 02:12:49.642686 6941 raft_consensus.cc:2726] T 91798b4bc2f74b9289425531937a78ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:49.642309 5105 tablet_bootstrap.cc:492] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
W20240503 02:12:49.642973 4746 leader_election.cc:334] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.643190 5105 ts_tablet_manager.cc:1367] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.015s
I20240503 02:12:49.643254 4746 leader_election.cc:302] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.643524 4746 raft_consensus.cc:2670] T 16b3d1e7ce83425db0bd56457d9f2544 P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:49.643934 5105 raft_consensus.cc:348] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
W20240503 02:12:49.644119 4746 consensus_peers.cc:485] T fe75d31e5da645be9ccdde4d0a4f9e05 P acc6d33969e74c209310005ae911c405 -> Peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Couldn't send request to peer 1f50719d94564844b15e1a9211d48a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:49.647287 4746 leader_election.cc:334] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355): Network error: Client connection negotiation failed: client connection to 127.4.17.66:35355: connect: Connection refused (error 111)
I20240503 02:12:49.647397 4746 leader_election.cc:302] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acc6d33969e74c209310005ae911c405; no voters: 1f50719d94564844b15e1a9211d48a81, f4f9f079230a468f90e5e3a58f254b91
W20240503 02:12:49.647557 4746 raft_consensus.cc:2670] T 5b91a3f3cad74f6aa6035a013cca651f P acc6d33969e74c209310005ae911c405: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:12:49.644165 5105 raft_consensus.cc:374] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.648362 5105 raft_consensus.cc:729] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc6d33969e74c209310005ae911c405, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.648587 5105 consensus_queue.cc:260] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:49.649210 5105 ts_tablet_manager.cc:1398] T 6624e145aac2412d98acf847e9e9c8b8 P acc6d33969e74c209310005ae911c405: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:49.649514 5105 tablet_bootstrap.cc:492] T c919bda1eaaa4e52828aac3ef337f3e5 P acc6d33969e74c209310005ae911c405: Bootstrap starting.
I20240503 02:12:49.651032 5105 tablet_bootstrap.cc:654] T c919bda1eaaa4e52828aac3ef337f3e5 P acc6d33969e74c209310005ae911c405: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.660439 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a915d760ba14a31895a5dc7ec5b34fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:27 UTC) from {username='slave'} at 127.0.0.1:39380
I20240503 02:12:49.663491 5159 ts_tablet_manager.cc:1888] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:49.663633 5159 log.cc:1196] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/17a24b5c83b649e79644b08253d851d6
I20240503 02:12:49.663925 5159 ts_tablet_manager.cc:1909] T 17a24b5c83b649e79644b08253d851d6 P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:49.665025 5159 tablet_replica.cc:330] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: stopping tablet replica
I20240503 02:12:49.665272 5159 raft_consensus.cc:2229] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:49.665424 5159 raft_consensus.cc:2258] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:49.666373 5105 tablet_bootstrap.cc:492] T c919bda1eaaa4e52828aac3ef337f3e5 P acc6d33969e74c209310005ae911c405: No bootstrap required, opened a new log
I20240503 02:12:49.666512 5105 ts_tablet_manager.cc:1367] T c919bda1eaaa4e52828aac3ef337f3e5 P acc6d33969e74c209310005ae911c405: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:12:49.666998 5159 ts_tablet_manager.cc:1875] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:49.670290 4259 catalog_manager.cc:4830] TS acc6d33969e74c209310005ae911c405 (127.4.17.67:35993): tablet 17a24b5c83b649e79644b08253d851d6 (table default.Table_7fe2fd4286f4411aad562211e4bd77ae [id=2073875662ee44ca99e5837ba4ace7ca]) successfully deleted
I20240503 02:12:49.670579 4853 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:50.082088 6814 raft_consensus.cc:482] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:50.084162 6814 raft_consensus.cc:504] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } }
I20240503 02:12:50.085093 6814 leader_election.cc:288] T b23be1160c454c139128aff5a960d183 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:50.086275 6941 raft_consensus.cc:482] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.086467 6941 raft_consensus.cc:504] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:50.087168 6941 leader_election.cc:288] T a5f8d331a718475a9f56dc7f3ec0e6b9 P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837), 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355)
I20240503 02:12:50.090318 5159 ts_tablet_manager.cc:1888] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:12:50.090489 5159 log.cc:1196] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/5a915d760ba14a31895a5dc7ec5b34fd
I20240503 02:12:50.090847 5159 ts_tablet_manager.cc:1909] T 5a915d760ba14a31895a5dc7ec5b34fd P acc6d33969e74c209310005ae911c405: Deleting consensus metadata
I20240503 02:12:50.097641 7105 raft_consensus.cc:482] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4f9f079230a468f90e5e3a58f254b91)
I20240503 02:12:50.097828 7105 raft_consensus.cc:504] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f50719d94564844b15e1a9211d48a81" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 35355 } } peers { permanent_uuid: "f4f9f079230a468f90e5e3a58f254b91" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 39837 } } peers { permanent_uuid: "acc6d33969e74c209310005ae911c405" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 35993 } }
I20240503 02:12:50.098554 7105 leader_election.cc:288] T 9a8e996d76be4dda876f4645df56c1ce P acc6d33969e74c209310005ae911c405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f50719d94564844b15e1a9211d48a81 (127.4.17.66:35355), f4f9f079230a468f90e5e3a58f254b91 (127.4.17.65:39837)
I20240503 02:12:50.100690 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4730
I20240503 02:12:50.155483 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 5268
I20240503 02:12:50.162433 5501 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:50.220196 4262 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1c084ab3fedb46d4992f5af2ceede06b (table default.Table_003ced126c26439d80b384dd00c1b8a7 [id=b655253e9eea42ac951931e69dd2c27c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c084ab3fedb46d4992f5af2ceede06b consensus state has no leader
I20240503 02:12:50.356365 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 5268
I20240503 02:12:50.394966 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 4241
I20240503 02:12:50.396001 4305 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:50.635836 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 4241
I20240503 02:12:50.682149 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 5937
I20240503 02:12:50.689688 6278 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:50.839773 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 5937
2024-05-03T02:12:50Z chronyd exiting
I20240503 02:12:50.881326 4165 master-stress-test.cc:586] Tables created: 556
I20240503 02:12:50.881433 4165 master-stress-test.cc:587] Tables altered: 276
I20240503 02:12:50.881480 4165 master-stress-test.cc:588] Tables deleted: 243
I20240503 02:12:50.881513 4165 master-stress-test.cc:589] Tablets replaced: 66
I20240503 02:12:50.881556 4165 master-stress-test.cc:590] Masters restarted: 3
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (39730 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20240503 02:12:51.041597 4165 test_util.cc:268] Using random seed: 1102623012
2024-05-03T02:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:51Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskIVKwFe/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskIVKwFe/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/security.properties
2024-05-03 02:12:57: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskIVKwFe/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskIVKwFe/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20240503 02:13:06.904219 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--webserver_interface=127.4.17.126
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:07.260735 7355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:07.261180 7355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:07.261340 7355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:07.269639 7355 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:07.269783 7355 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:07.269867 7355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:07.269953 7355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:07.270031 7355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:07.270133 7355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:07.279976 7355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:07.281548 7355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:07.283573 7355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:07.297852 7361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:07.298439 7360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:07.299436 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:07.300597 7355 server_base.cc:1034] running on GCE node
I20240503 02:13:07.301558 7355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:07.303185 7355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:07.304455 7355 hybrid_clock.cc:648] HybridClock initialized: now 1714702387304380 us; error 49 us; skew 500 ppm
I20240503 02:13:07.304862 7355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:07.314046 7355 webserver.cc:460] Webserver started at http://127.4.17.126:39555/ using document root <none> and password file <none>
I20240503 02:13:07.314940 7355 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:07.315086 7355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:07.315449 7355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:07.319690 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/instance:
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.320636 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal/instance:
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.326772 7355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20240503 02:13:07.330889 7369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:07.332530 7355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:07.332744 7355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.332996 7355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:07.354077 7355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:07.355105 7355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:07.355475 7355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:07.375828 7355 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:42485
I20240503 02:13:07.375857 7420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:42485 every 8 connection(s)
I20240503 02:13:07.377454 7355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:13:07.381052 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7355
I20240503 02:13:07.381289 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal/instance
I20240503 02:13:07.382105 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.382761 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--webserver_interface=127.4.17.125
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:13:07.392194 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:07.411559 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:13:07.413098 7371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:07.415016 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:07.464429 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } attempt: 1
W20240503 02:13:07.466637 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:07.534930 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } attempt: 2
W20240503 02:13:07.537236 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:07.628634 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } attempt: 3
W20240503 02:13:07.630899 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:07.745795 7422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:07.746109 7422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:07.746249 7422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:07.755364 7422 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:07.755501 7422 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:07.755555 7422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:07.755645 7422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:07.755720 7422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:07.755805 7422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:07.765781 7422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.17.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:07.767258 7422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:07.769160 7422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:13:07.774209 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } attempt: 4
W20240503 02:13:07.776859 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:07.781772 7430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:07.783033 7433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:07.783969 7431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:07.785317 7422 server_base.cc:1034] running on GCE node
I20240503 02:13:07.786216 7422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:07.787828 7422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:07.789079 7422 hybrid_clock.cc:648] HybridClock initialized: now 1714702387788996 us; error 53 us; skew 500 ppm
I20240503 02:13:07.789508 7422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:07.792203 7422 webserver.cc:460] Webserver started at http://127.4.17.125:46239/ using document root <none> and password file <none>
I20240503 02:13:07.793081 7422 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:07.793241 7422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:07.793587 7422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:07.796442 7422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/instance:
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.797300 7422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal/instance:
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.803416 7422 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240503 02:13:07.807699 7439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:07.809492 7422 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:07.809701 7422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:07.809943 7422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:07.837142 7422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:07.838161 7422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:07.838500 7422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:07.859862 7422 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.125:45323
I20240503 02:13:07.859905 7490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.125:45323 every 8 connection(s)
I20240503 02:13:07.861438 7422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
I20240503 02:13:07.865118 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7422
I20240503 02:13:07.865335 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal/instance
I20240503 02:13:07.865756 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.866797 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:34141
--webserver_interface=127.4.17.124
--webserver_port=0
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:13:07.875389 7491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:07.891155 7491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:07.901808 7491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:13:07.902961 7441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:07.904610 7491 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:07.953969 7491 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 1
W20240503 02:13:07.956413 7491 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.024721 7491 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 2
W20240503 02:13:08.026980 7491 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.076203 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } attempt: 5
I20240503 02:13:08.083478 7421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:13:08.085312 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.118248 7491 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 3
W20240503 02:13:08.120407 7491 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.136571 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 1
W20240503 02:13:08.138777 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.207052 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 2
W20240503 02:13:08.209270 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:08.224749 7492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:08.225165 7492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:08.225298 7492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:08.233037 7492 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:08.233166 7492 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:08.233246 7492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:08.233341 7492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:08.233415 7492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:08.233502 7492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:08.243198 7492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.17.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:08.244665 7492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:08.246503 7492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:08.258553 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:08.259152 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:08.260125 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:08.261269 7492 server_base.cc:1034] running on GCE node
I20240503 02:13:08.262164 7492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:08.263705 7492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:08.263643 7491 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 4
I20240503 02:13:08.264997 7492 hybrid_clock.cc:648] HybridClock initialized: now 1714702388264888 us; error 54 us; skew 500 ppm
W20240503 02:13:08.265476 7491 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.265493 7492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:08.267956 7492 webserver.cc:460] Webserver started at http://127.4.17.124:34607/ using document root <none> and password file <none>
I20240503 02:13:08.268707 7492 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:08.268837 7492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:08.269169 7492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:08.272136 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/instance:
uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
format_stamp: "Formatted at 2024-05-03 02:13:08 on dist-test-slave-ddxx"
I20240503 02:13:08.273013 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal/instance:
uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
format_stamp: "Formatted at 2024-05-03 02:13:08 on dist-test-slave-ddxx"
I20240503 02:13:08.278599 7492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:08.282274 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:08.283738 7492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 02:13:08.283914 7492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
format_stamp: "Formatted at 2024-05-03 02:13:08 on dist-test-slave-ddxx"
I20240503 02:13:08.284106 7492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:08.304657 7492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:08.305617 7492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:08.305922 7492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:08.315554 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 3
W20240503 02:13:08.317966 7421 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:08.324862 7492 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.124:34141
I20240503 02:13:08.324901 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.124:34141 every 8 connection(s)
I20240503 02:13:08.326354 7492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
I20240503 02:13:08.326754 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7492
I20240503 02:13:08.326929 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal/instance
I20240503 02:13:08.330864 7561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.337936 7561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:08.351222 7561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:08.357884 7561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:13:08.368160 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Bootstrap starting.
I20240503 02:13:08.371376 7561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.372467 7561 log.cc:824] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:08.375533 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: No bootstrap required, opened a new log
I20240503 02:13:08.382185 7561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.382501 7561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.382628 7561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.383289 7561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.384235 7566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.384603 7566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.385251 7561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:08.399966 7566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.400249 7566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
W20240503 02:13:08.401744 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.403733 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.403044 7566 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
I20240503 02:13:08.394399 7492 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: OK, instance_id { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" instance_seqno: 1714702387370161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:13:08.404392 7492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:13:08.416373 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.416692 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.432602 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.433038 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.444717 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.444876 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.456573 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.456808 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.469391 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.469755 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.481326 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.481536 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:13:08.493333 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.493490 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.495194 7421 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 4
W20240503 02:13:08.505260 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.505398 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.508065 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap starting.
I20240503 02:13:08.512450 7421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Neither blocks nor log segments found. Creating new log.
W20240503 02:13:08.517104 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.517264 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.522643 7421 log.cc:824] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:08.526181 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: No bootstrap required, opened a new log
W20240503 02:13:08.528676 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.528808 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.537636 7421 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.538065 7421 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.538220 7421 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.539031 7421 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
W20240503 02:13:08.540518 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.540652 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.540738 7584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.541146 7584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.541918 7421 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:08.541960 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293" is_pre_election: true
I20240503 02:13:08.542640 7396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 0.
I20240503 02:13:08.543373 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293; no voters:
I20240503 02:13:08.543891 7566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.544045 7566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.544147 7566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.547443 7566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.548795 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293"
I20240503 02:13:08.549063 7396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.549119 7566 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 1 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:08.552145 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.552263 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.552733 7396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 1.
I20240503 02:13:08.553282 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293; no voters:
I20240503 02:13:08.553874 7566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.557278 7566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Running, Role: LEADER
W20240503 02:13:08.558229 7515 tablet.cc:2369] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:08.558153 7566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.564693 7491 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } attempt: 5
I20240503 02:13:08.572391 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.572810 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
W20240503 02:13:08.574468 7580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:13:08.574573 7580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:13:08.576114 7592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:08.580457 7592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:08.583254 7491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap starting.
I20240503 02:13:08.591022 7491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.593955 7491 log.cc:824] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:08.606864 7491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: No bootstrap required, opened a new log
W20240503 02:13:08.608863 7375 tablet.cc:2369] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:08.609472 7396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Refusing update from remote peer 0fefe1a587244be2b2dec28f7fa3f9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.610599 7588 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.620308 7491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.620723 7491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.620895 7491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.621735 7491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:08.623183 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.623739 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.624660 7491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:08.624693 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" is_pre_election: true
I20240503 02:13:08.625478 7464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.624660 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
I20240503 02:13:08.631556 7464 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Refusing update from remote peer 0fefe1a587244be2b2dec28f7fa3f9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.633728 7588 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.650439 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.650883 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.653260 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.653561 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:08.654238 7595 mvcc.cc:205] Tried to move back new op lower bound from 7023420983724187648 to 7023420983540137984. Current Snapshot: MvccSnapshot[applied={T|T < 7023420983724187648 or (T in {7023420983724187648})}]
I20240503 02:13:08.655128 7592 catalog_manager.cc:1347] Generated new cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:08.655256 7592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:08.666671 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.666936 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.672601 7566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.672986 7566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:08.673470 7584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.673784 7584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.680187 7584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:08.680454 7584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:08.688681 7592 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:13:08.690652 7592 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:08.696218 7627 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 02:13:08.698129 7625 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 02:13:08.698580 7422 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: OK, instance_id { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" instance_seqno: 1714702387370161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 02:13:08.714833 7422 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:08.735119 7592 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Generated new TSK 0
I20240503 02:13:08.736531 7592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:08.738899 7592 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:08.740202 7592 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:08.742769 7625 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:08.744520 7625 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: loading cluster ID for follower catalog manager: success
I20240503 02:13:08.758129 7627 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:08.758281 7627 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: loading cluster ID for follower catalog manager: success
I20240503 02:13:08.760655 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.65:0
--local_ip_for_outbound_sockets=127.4.17.65
--webserver_interface=127.4.17.65
--webserver_port=0
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:13:08.764307 7627 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: acquiring CA information for follower catalog manager: success
I20240503 02:13:08.748865 7625 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: acquiring CA information for follower catalog manager: success
I20240503 02:13:08.767432 7625 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:08.780556 7627 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240503 02:13:09.285193 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:09.285681 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:09.285867 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:09.298959 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:09.299253 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.65
I20240503 02:13:09.314642 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.17.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.65
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:09.316560 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:09.318965 7630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:09.335911 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.336932 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.347393 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:09.348505 7630 server_base.cc:1034] running on GCE node
I20240503 02:13:09.349350 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:09.350636 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:09.351948 7630 hybrid_clock.cc:648] HybridClock initialized: now 1714702389351856 us; error 59 us; skew 500 ppm
I20240503 02:13:09.352288 7630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:09.354876 7630 webserver.cc:460] Webserver started at http://127.4.17.65:42949/ using document root <none> and password file <none>
I20240503 02:13:09.355643 7630 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:09.355782 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:09.356076 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:09.359118 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data/instance:
uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.359906 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.365621 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 02:13:09.369535 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.371090 7630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:13:09.371255 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal
uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.371459 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:09.442159 7630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:09.443078 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:09.443396 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:09.444411 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:09.446110 7630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:09.446183 7630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.446308 7630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:09.446369 7630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.477980 7630 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.65:45901
I20240503 02:13:09.478060 7756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.65:45901 every 8 connection(s)
I20240503 02:13:09.479339 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/data/info.pb
I20240503 02:13:09.480865 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7630
I20240503 02:13:09.481119 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal/instance
I20240503 02:13:09.490268 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.66:0
--local_ip_for_outbound_sockets=127.4.17.66
--webserver_interface=127.4.17.66
--webserver_port=0
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:13:09.518029 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:09.518440 7758 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:09.519217 7757 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:09.519383 7757 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:09.519737 7758 heartbeater.cc:511] Master 127.4.17.126:42485 requested a full tablet report, sending...
I20240503 02:13:09.519953 7757 heartbeater.cc:511] Master 127.4.17.124:34141 requested a full tablet report, sending...
I20240503 02:13:09.522213 7386 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:09.522213 7526 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:09.523831 7526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.65:33489
I20240503 02:13:09.526646 7759 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:09.526793 7759 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:09.527382 7759 heartbeater.cc:511] Master 127.4.17.125:45323 requested a full tablet report, sending...
I20240503 02:13:09.529004 7456 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:09.637615 7768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:09.637859 7768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:09.639338 7768 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:09.640058 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293" is_pre_election: true
I20240503 02:13:09.640112 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" is_pre_election: true
I20240503 02:13:09.641022 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:09.641434 7768 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:09.674288 7769 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:09.674549 7769 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:09.675964 7769 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:09.676581 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" is_pre_election: true
I20240503 02:13:09.676656 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" is_pre_election: true
I20240503 02:13:09.676859 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2855cc86e414615a261b7a7d4bd3293 in term 1.
I20240503 02:13:09.677479 7371 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9
I20240503 02:13:09.677899 7769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:09.678122 7769 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:09.678221 7769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.680399 7769 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:09.681187 7769 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 2 election: Requested vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:09.681733 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
I20240503 02:13:09.681790 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
I20240503 02:13:09.681944 7465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.683952 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2855cc86e414615a261b7a7d4bd3293 in term 2.
I20240503 02:13:09.684381 7371 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9
I20240503 02:13:09.684762 7769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:09.685285 7769 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 LEADER]: Becoming Leader. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Running, Role: LEADER
I20240503 02:13:09.685788 7769 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:09.687422 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2855cc86e414615a261b7a7d4bd3293. Latest consensus state: current_term: 2 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:09.687776 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:09.688606 7771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:09.690788 7771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:09.691644 7771 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:09.691717 7771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:09.692767 7771 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:09.693594 7771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Loaded TSK: 0
I20240503 02:13:09.694383 7771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:09.695124 7771 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:09.695861 7771 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:09.703472 7465 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 2 FOLLOWER]: Rejecting Update request from peer 0fefe1a587244be2b2dec28f7fa3f9d9 for earlier term 1. Current term is 2. Ops: []
I20240503 02:13:09.704169 7772 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:09.704835 7773 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:09.704993 7773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Running, Role: LEADER
I20240503 02:13:09.705547 7773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:09.706389 7773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.712136 7580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240503 02:13:09.865271 7766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:09.865739 7766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:09.865938 7766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:09.875468 7766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:09.875720 7766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.66
I20240503 02:13:09.886091 7766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.17.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.66
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:09.887806 7766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:09.889978 7766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:09.904460 7778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.905580 7779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.906687 7781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:09.907811 7766 server_base.cc:1034] running on GCE node
I20240503 02:13:09.908576 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:09.909708 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:09.910966 7766 hybrid_clock.cc:648] HybridClock initialized: now 1714702389910892 us; error 52 us; skew 500 ppm
I20240503 02:13:09.911458 7766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:09.914165 7766 webserver.cc:460] Webserver started at http://127.4.17.66:34497/ using document root <none> and password file <none>
I20240503 02:13:09.915014 7766 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:09.915159 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:09.915500 7766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:09.918418 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data/instance:
uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.919186 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.925472 7766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:13:09.929665 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.931537 7766 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:13:09.931739 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal
uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-ddxx"
I20240503 02:13:09.931965 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:09.962172 7766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:09.963076 7766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:09.963371 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:09.964566 7766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:09.966220 7766 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:09.966298 7766 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.966388 7766 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:09.966431 7766 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.999269 7766 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.66:33615
I20240503 02:13:09.999352 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.66:33615 every 8 connection(s)
I20240503 02:13:10.000735 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/data/info.pb
I20240503 02:13:10.009852 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7766
I20240503 02:13:10.010090 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal/instance
I20240503 02:13:10.017403 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.17.67:0
--local_ip_for_outbound_sockets=127.4.17.67
--webserver_interface=127.4.17.67
--webserver_port=0
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 02:13:10.019336 7901 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:10.019652 7902 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:10.019774 7901 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.019838 7902 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.021039 7900 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:10.021111 7901 heartbeater.cc:511] Master 127.4.17.126:42485 requested a full tablet report, sending...
I20240503 02:13:10.021133 7902 heartbeater.cc:511] Master 127.4.17.125:45323 requested a full tablet report, sending...
I20240503 02:13:10.021212 7900 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.021831 7900 heartbeater.cc:511] Master 127.4.17.124:34141 requested a full tablet report, sending...
I20240503 02:13:10.022856 7386 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:10.023191 7456 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:10.024518 7386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.66:43905
I20240503 02:13:10.026417 7526 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:10.027609 7757 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:10.151120 7773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.151378 7773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.152339 7773 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
I20240503 02:13:10.152904 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293" is_pre_election: true
I20240503 02:13:10.152998 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" is_pre_election: true
I20240503 02:13:10.153242 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 2.
I20240503 02:13:10.153682 7511 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:10.153960 7773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:13:10.154088 7773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.154176 7773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.156975 7773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.157447 7536 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Rejecting Update request from peer f2855cc86e414615a261b7a7d4bd3293 for earlier term 2. Current term is 3. Ops: []
I20240503 02:13:10.157763 7773 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 3 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
I20240503 02:13:10.158329 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
I20240503 02:13:10.158587 7465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.158488 7770 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.158752 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293"
I20240503 02:13:10.158938 7770 raft_consensus.cc:3032] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:10.159057 7770 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Running, Role: LEADER
I20240503 02:13:10.159384 7770 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.159958 7770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.161551 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 3.
I20240503 02:13:10.162093 7511 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:10.163167 7773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:10.163359 7773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 LEADER]: Becoming Leader. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Running, Role: LEADER
I20240503 02:13:10.163750 7773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.165423 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 3 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:10.165699 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:10.166268 7909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:10.167833 7909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:10.168709 7909 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:10.168782 7909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:10.170085 7909 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:10.170866 7909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Loaded TSK: 0
I20240503 02:13:10.171506 7909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:10.172194 7909 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:10.172904 7909 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
W20240503 02:13:10.395038 7907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:10.395391 7907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:10.395556 7907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:10.403455 7907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:10.403719 7907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.17.67
I20240503 02:13:10.413254 7907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.17.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.17.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.17.67
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:10.414803 7907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:10.416697 7907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:10.431295 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.432163 7917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.433076 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:10.434048 7907 server_base.cc:1034] running on GCE node
I20240503 02:13:10.434777 7907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:10.435909 7907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:10.437366 7907 hybrid_clock.cc:648] HybridClock initialized: now 1714702390437263 us; error 68 us; skew 500 ppm
I20240503 02:13:10.437882 7907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:10.441295 7907 webserver.cc:460] Webserver started at http://127.4.17.67:37157/ using document root <none> and password file <none>
I20240503 02:13:10.442356 7907 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:10.442559 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:10.442839 7907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:10.445737 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data/instance:
uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-ddxx"
I20240503 02:13:10.446493 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/instance:
uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-ddxx"
I20240503 02:13:10.452248 7907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:13:10.456236 7923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.458035 7907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:13:10.458258 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal
uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-ddxx"
I20240503 02:13:10.458482 7907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:10.473902 7907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:10.475044 7907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:10.475420 7907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:10.476568 7907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:10.478497 7907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:10.478587 7907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.478737 7907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:10.478796 7907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.512851 7907 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.67:41095
I20240503 02:13:10.512887 8035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.67:41095 every 8 connection(s)
I20240503 02:13:10.514289 7907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/data/info.pb
I20240503 02:13:10.521207 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 7907
I20240503 02:13:10.521484 4165 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/instance
I20240503 02:13:10.528776 7758 heartbeater.cc:503] Master 127.4.17.126:42485 was elected leader, sending a full tablet report...
I20240503 02:13:10.529187 7901 heartbeater.cc:503] Master 127.4.17.126:42485 was elected leader, sending a full tablet report...
I20240503 02:13:10.535048 8037 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:10.535429 8037 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.536629 8036 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:10.536685 8037 heartbeater.cc:511] Master 127.4.17.126:42485 requested a full tablet report, sending...
I20240503 02:13:10.536818 8036 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.537351 8036 heartbeater.cc:511] Master 127.4.17.124:34141 requested a full tablet report, sending...
I20240503 02:13:10.538473 7525 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:10.538739 7384 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:10.540238 7525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.17.67:48885
I20240503 02:13:10.544857 8038 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:10.545025 8038 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.545377 8038 heartbeater.cc:511] Master 127.4.17.125:45323 requested a full tablet report, sending...
I20240503 02:13:10.546315 7456 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:10.554128 4165 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 02:13:10.577315 7524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_fa460008f83a4dc499b1ee88ee0e7925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.577370 7526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_cba0c48d8df441209b09207d54435b46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.577315 7523 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_bd10243fcd8e4436952edc7daf12e223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.578441 7525 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_82fc16f07bdb4d24bb75d0b04ffe9c29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.580667 7522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_81067e0e8faa4902a7f68b688523e3e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.581238 7521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.582887 7520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_ddb14bf51d034e819a0d6b707439b04c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.583962 7519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_0ec6c406990740d085be8679baa773d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.586056 7518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_830f8efb3147411b8c2863df686abe6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.586792 7517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58196:
name: "default.Table_6c742fade1b9400f93b9e39783dbecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:13:10.587741 7526 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cba0c48d8df441209b09207d54435b46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:10.609866 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.610071 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.610728 7770 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:10.611380 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" is_pre_election: true
I20240503 02:13:10.611620 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2855cc86e414615a261b7a7d4bd3293 in term 3.
I20240503 02:13:10.612129 7371 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters:
I20240503 02:13:10.612589 7770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 02:13:10.612694 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.612805 7770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:10.613437 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" is_pre_election: true
I20240503 02:13:10.615775 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.616657 7770 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 4 election: Requested vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:10.617745 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
I20240503 02:13:10.617836 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2855cc86e414615a261b7a7d4bd3293" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
I20240503 02:13:10.617993 7465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:10.620975 7465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2855cc86e414615a261b7a7d4bd3293 in term 4.
I20240503 02:13:10.621929 7371 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9
I20240503 02:13:10.622593 7770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:13:10.622994 7770 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 LEADER]: Becoming Leader. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Running, Role: LEADER
I20240503 02:13:10.623288 7770 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:10.624931 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2855cc86e414615a261b7a7d4bd3293. Latest consensus state: current_term: 4 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:10.625237 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:10.626164 8083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:10.629225 8083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:10.631031 8083 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:10.631242 8083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:10.633282 8083 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:10.634833 8083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Loaded TSK: 0
I20240503 02:13:10.635735 8083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:10.636253 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 7492
I20240503 02:13:10.637117 8083 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:10.637759 7556 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:10.638092 8083 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:10.806089 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 7492
I20240503 02:13:10.806075 7465 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Rejecting Update request from peer 0fefe1a587244be2b2dec28f7fa3f9d9 for earlier term 3. Current term is 4. Ops: []
I20240503 02:13:10.806136 7396 raft_consensus.cc:1226] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 LEADER]: Rejecting Update request from peer 0fefe1a587244be2b2dec28f7fa3f9d9 for earlier term 3. Current term is 4. Ops: []
W20240503 02:13:10.835170 8046 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.124:34141)
W20240503 02:13:10.838196 7443 connection.cc:535] server connection from 127.0.0.1:59604 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:10.838912 7370 connection.cc:535] server connection from 127.0.0.1:36488 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:10.844000 8050 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:10.836508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.17.124:34141 (local address 127.0.0.1:58208)
0503 02:13:10.837167 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:10.837190 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:10.837316 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:10.842448 (+ 5132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":489}
I20240503 02:13:10.840621 7385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_830f8efb3147411b8c2863df686abe6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.841425 7380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_fa460008f83a4dc499b1ee88ee0e7925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.841805 7378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_81067e0e8faa4902a7f68b688523e3e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.841084 7382 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_cba0c48d8df441209b09207d54435b46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.841662 7379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_bd10243fcd8e4436952edc7daf12e223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.841269 7381 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_82fc16f07bdb4d24bb75d0b04ffe9c29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.840685 7377 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.840222 7386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_0ec6c406990740d085be8679baa773d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.842063 7384 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_6c742fade1b9400f93b9e39783dbecc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:10.840221 7383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_ddb14bf51d034e819a0d6b707439b04c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:13:10.857105 7385 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_830f8efb3147411b8c2863df686abe6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:13:10.857105 7380 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fa460008f83a4dc499b1ee88ee0e7925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:11.034041 7900 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:11.035715 7757 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
W20240503 02:13:11.036998 7900 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:11.038504 7757 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:11.042953 8036 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
W20240503 02:13:11.044158 8036 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:11.096884 7385 catalog_manager.cc:2145] created HMS catalog entry for table [id=c74da947f58744958c70d8bb9a9d756c]
I20240503 02:13:11.117198 7465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Refusing update from remote peer f2855cc86e414615a261b7a7d4bd3293: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 02:13:11.121564 7769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:11.129173 7371 consensus_peers.cc:485] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 -> Peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Couldn't send request to peer 0fefe1a587244be2b2dec28f7fa3f9d9. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:11.142973 7380 catalog_manager.cc:2145] created HMS catalog entry for table [id=f86e3f6464b74f0d84e2c1476e9fb583]
W20240503 02:13:11.146420 7378 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_81067e0e8faa4902a7f68b688523e3e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:11.146919 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2855cc86e414615a261b7a7d4bd3293. Latest consensus state: current_term: 4 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:11.147375 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:13:11.147763 7382 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cba0c48d8df441209b09207d54435b46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:13:11.148782 7379 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bd10243fcd8e4436952edc7daf12e223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:13:11.149652 7381 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:11.150398 8088 mvcc.cc:205] Tried to move back new op lower bound from 7023420993941884928 to 7023420988157222912. Current Snapshot: MvccSnapshot[applied={T|T < 7023420993941884928}]
W20240503 02:13:11.150646 7377 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f1ff227bd5304d93a4c1ef884df9c987 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:13:11.151841 7386 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0ec6c406990740d085be8679baa773d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:11.152027 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:11.152266 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: LEADER
W20240503 02:13:11.152734 7384 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6c742fade1b9400f93b9e39783dbecc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 02:13:11.153721 7383 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ddb14bf51d034e819a0d6b707439b04c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:11.161653 7385 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_830f8efb3147411b8c2863df686abe6c" } new_table_name: "default.Table_46c477c0b6584ff8a2279dc7fba52e9c" modify_external_catalogs: true
I20240503 02:13:11.175410 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:11.175720 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:11.175426 8096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f2855cc86e414615a261b7a7d4bd3293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:11.175853 8096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:11.179878 7380 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_fa460008f83a4dc499b1ee88ee0e7925" } new_table_name: "default.Table_93b16ebe2e78419f85f3016868132e3e" modify_external_catalogs: true
I20240503 02:13:11.204458 7378 catalog_manager.cc:2145] created HMS catalog entry for table [id=4ef7334f794b4e53b19ff587daf1455d]
I20240503 02:13:11.243090 7378 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_81067e0e8faa4902a7f68b688523e3e4" } new_table_name: "default.Table_46f2cd6cae304057bb0f555ab6d065fc" modify_external_catalogs: true
I20240503 02:13:11.250502 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 2906301538e04dfb962fff2381023ba0 (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.251833 7691 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dca752b6947f8a891a9af6a58fdf4 (DEFAULT_TABLE table=default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.253736 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dca752b6947f8a891a9af6a58fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.257231 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 05406776807c49419ee4067df0b023a9 (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.257642 7692 tablet_service.cc:1467] Processing CreateTablet for tablet 2906301538e04dfb962fff2381023ba0 (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.257859 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05406776807c49419ee4067df0b023a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.267215 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906301538e04dfb962fff2381023ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.266520 7970 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dca752b6947f8a891a9af6a58fdf4 (DEFAULT_TABLE table=default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.268476 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dca752b6947f8a891a9af6a58fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.269081 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906301538e04dfb962fff2381023ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.269410 7969 tablet_service.cc:1467] Processing CreateTablet for tablet 05406776807c49419ee4067df0b023a9 (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.269999 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05406776807c49419ee4067df0b023a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.271265 7971 tablet_service.cc:1467] Processing CreateTablet for tablet 2906301538e04dfb962fff2381023ba0 (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.271931 7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906301538e04dfb962fff2381023ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.274207 7834 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dca752b6947f8a891a9af6a58fdf4 (DEFAULT_TABLE table=default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.274274 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 05406776807c49419ee4067df0b023a9 (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.274888 7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dca752b6947f8a891a9af6a58fdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.277446 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05406776807c49419ee4067df0b023a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.294574 8106 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.301286 8106 tablet_bootstrap.cc:654] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.303354 8106 log.cc:824] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:11.311126 8108 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.312247 8109 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.322815 8109 tablet_bootstrap.cc:654] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.325590 8108 tablet_bootstrap.cc:654] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.328434 8108 log.cc:824] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:11.329303 8106 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.329792 8106 ts_tablet_manager.cc:1367] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.004s
I20240503 02:13:11.329895 8109 log.cc:824] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:11.342059 8108 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.342586 8108 ts_tablet_manager.cc:1367] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.003s
I20240503 02:13:11.345254 8106 raft_consensus.cc:348] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.346369 8106 raft_consensus.cc:374] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.346565 8106 raft_consensus.cc:729] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.346827 8109 tablet_bootstrap.cc:492] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.347206 8109 ts_tablet_manager.cc:1367] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.000s
I20240503 02:13:11.347591 8106 consensus_queue.cc:260] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.357579 8106 ts_tablet_manager.cc:1398] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.028s user 0.013s sys 0.007s
I20240503 02:13:11.360649 8037 heartbeater.cc:503] Master 127.4.17.126:42485 was elected leader, sending a full tablet report...
I20240503 02:13:11.366362 8108 raft_consensus.cc:348] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.367305 8106 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.368571 8109 raft_consensus.cc:348] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.369257 8108 raft_consensus.cc:374] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.369457 8108 raft_consensus.cc:729] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.369604 8106 tablet_bootstrap.cc:654] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.370555 8108 consensus_queue.cc:260] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.370977 8109 raft_consensus.cc:374] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.371423 8109 raft_consensus.cc:729] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.372382 8109 consensus_queue.cc:260] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.374533 8108 ts_tablet_manager.cc:1398] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.032s user 0.010s sys 0.002s
I20240503 02:13:11.374926 7758 heartbeater.cc:503] Master 127.4.17.126:42485 was elected leader, sending a full tablet report...
I20240503 02:13:11.378465 8108 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.380458 8108 tablet_bootstrap.cc:654] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.384459 8109 ts_tablet_manager.cc:1398] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.037s user 0.013s sys 0.001s
I20240503 02:13:11.386268 7901 heartbeater.cc:503] Master 127.4.17.126:42485 was elected leader, sending a full tablet report...
I20240503 02:13:11.389390 8109 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.392760 8109 tablet_bootstrap.cc:654] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.395802 8108 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.396085 8108 ts_tablet_manager.cc:1367] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20240503 02:13:11.396498 8106 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.396730 8106 ts_tablet_manager.cc:1367] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.007s
I20240503 02:13:11.397346 8108 raft_consensus.cc:348] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.398778 8108 raft_consensus.cc:374] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.398964 8108 raft_consensus.cc:729] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.398937 8106 raft_consensus.cc:348] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.399175 8106 raft_consensus.cc:374] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.399279 8106 raft_consensus.cc:729] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.399504 8108 consensus_queue.cc:260] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.399556 8106 consensus_queue.cc:260] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.400223 8108 ts_tablet_manager.cc:1398] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:11.401772 8108 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.403575 8108 tablet_bootstrap.cc:654] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.409336 8108 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.409890 8108 ts_tablet_manager.cc:1367] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 02:13:11.410549 8106 ts_tablet_manager.cc:1398] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:13:11.411811 8108 raft_consensus.cc:348] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.412967 8109 tablet_bootstrap.cc:492] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.413197 8109 ts_tablet_manager.cc:1367] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.003s
I20240503 02:13:11.414261 8106 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.414487 8109 raft_consensus.cc:348] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.414777 8109 raft_consensus.cc:374] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.414958 8109 raft_consensus.cc:729] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.415331 8109 consensus_queue.cc:260] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.416291 8106 tablet_bootstrap.cc:654] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.418821 8109 ts_tablet_manager.cc:1398] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:11.421722 8109 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.422022 8108 raft_consensus.cc:374] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.422221 8108 raft_consensus.cc:729] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.425199 8108 consensus_queue.cc:260] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.426223 8108 ts_tablet_manager.cc:1398] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 02:13:11.426963 8109 tablet_bootstrap.cc:654] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.435771 8109 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.436065 8106 tablet_bootstrap.cc:492] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.436293 8106 ts_tablet_manager.cc:1367] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20240503 02:13:11.436275 8109 ts_tablet_manager.cc:1367] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20240503 02:13:11.437312 8106 raft_consensus.cc:348] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.437605 8106 raft_consensus.cc:374] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.437388 8109 raft_consensus.cc:348] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.437693 8106 raft_consensus.cc:729] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.437923 8109 raft_consensus.cc:374] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.438066 8109 raft_consensus.cc:729] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.438021 8106 consensus_queue.cc:260] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.438555 8109 consensus_queue.cc:260] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.441193 8109 ts_tablet_manager.cc:1398] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:11.443527 8106 ts_tablet_manager.cc:1398] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240503 02:13:11.483546 7760 tablet.cc:2369] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:13:11.504967 7903 tablet.cc:2369] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:13:11.504 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_cba0c48d8df441209b09207d54435b46 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20240503 02:13:11.507246 7382 catalog_manager.cc:2141] Already present: failed to create HMS catalog entry for table [id=03b38bff293c4a6ead4aa4309aa9b7b4]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_cba0c48d8df441209b09207d54435b46 already exists)
I20240503 02:13:11.515530 8114 raft_consensus.cc:482] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.515859 8114 raft_consensus.cc:504] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.518142 8114 leader_election.cc:288] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
W20240503 02:13:11.519364 8039 tablet.cc:2369] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:11.529259 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906301538e04dfb962fff2381023ba0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:11.529956 7855 raft_consensus.cc:2454] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:11.531260 7645 leader_election.cc:302] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.531860 8114 raft_consensus.cc:2781] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.532045 8114 raft_consensus.cc:482] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.532171 8114 raft_consensus.cc:3037] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.536545 8114 raft_consensus.cc:504] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.538130 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906301538e04dfb962fff2381023ba0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:11.538391 7855 raft_consensus.cc:3037] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.542600 7855 raft_consensus.cc:2454] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:11.543437 7645 leader_election.cc:302] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.543960 8114 leader_election.cc:288] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Requested vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:11.546898 8114 raft_consensus.cc:2781] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.552783 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906301538e04dfb962fff2381023ba0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:11.553714 7991 raft_consensus.cc:2454] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:11.554423 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906301538e04dfb962fff2381023ba0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:11.554775 7990 raft_consensus.cc:3037] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.561319 7990 raft_consensus.cc:2454] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:11.562594 8114 raft_consensus.cc:686] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:11.563908 8114 consensus_queue.cc:237] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.575215 7382 master_service.cc:946] ReplaceTablet: received request to replace tablet 2906301538e04dfb962fff2381023ba0 from {username='slave'} at 127.0.0.1:36498
I20240503 02:13:11.582978 7379 catalog_manager.cc:2145] created HMS catalog entry for table [id=26947ead47f4427ba253ee462c99b657]
I20240503 02:13:11.594739 7379 master_service.cc:946] ReplaceTablet: received request to replace tablet 05406776807c49419ee4067df0b023a9 from {username='slave'} at 127.0.0.1:36498
I20240503 02:13:11.595229 7691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906301538e04dfb962fff2381023ba0 with delete_type TABLET_DATA_DELETED (replaced by tablet 27095a9813314d12804fa01ff5532b3c) from {username='slave'} at 127.0.0.1:33542
I20240503 02:13:11.597080 7382 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Sending DeleteTablet for 3 replicas of tablet 2906301538e04dfb962fff2381023ba0
I20240503 02:13:11.598282 7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906301538e04dfb962fff2381023ba0 with delete_type TABLET_DATA_DELETED (replaced by tablet 27095a9813314d12804fa01ff5532b3c) from {username='slave'} at 127.0.0.1:40404
I20240503 02:13:11.598547 7971 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906301538e04dfb962fff2381023ba0 with delete_type TABLET_DATA_DELETED (replaced by tablet 27095a9813314d12804fa01ff5532b3c) from {username='slave'} at 127.0.0.1:58384
I20240503 02:13:11.599574 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906301538e04dfb962fff2381023ba0 with delete_type TABLET_DATA_DELETED (replaced by tablet 27095a9813314d12804fa01ff5532b3c) from {username='slave'} at 127.0.0.1:33542
I20240503 02:13:11.604797 7382 catalog_manager.cc:6554] ReplaceTablet: tablet 2906301538e04dfb962fff2381023ba0 deleted and replaced by tablet 27095a9813314d12804fa01ff5532b3c
I20240503 02:13:11.605552 7382 master_service.cc:946] ReplaceTablet: received request to replace tablet 05406776807c49419ee4067df0b023a9 from {username='slave'} at 127.0.0.1:36498
I20240503 02:13:11.611168 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 7099ff89ceb544119b317319a4d0aa23 (DEFAULT_TABLE table=default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.611568 7969 tablet_service.cc:1467] Processing CreateTablet for tablet 7099ff89ceb544119b317319a4d0aa23 (DEFAULT_TABLE table=default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.611828 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099ff89ceb544119b317319a4d0aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.612035 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099ff89ceb544119b317319a4d0aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.613970 8126 tablet_replica.cc:330] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: stopping tablet replica
I20240503 02:13:11.614542 8126 raft_consensus.cc:2229] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.615139 8126 raft_consensus.cc:2258] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.618923 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 7099ff89ceb544119b317319a4d0aa23 (DEFAULT_TABLE table=default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.619412 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7099ff89ceb544119b317319a4d0aa23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.619999 8108 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.621258 8115 raft_consensus.cc:482] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.621429 8115 raft_consensus.cc:504] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.621728 8108 tablet_bootstrap.cc:654] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.623867 8115 leader_election.cc:288] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:11.625736 7692 tablet_service.cc:1467] Processing CreateTablet for tablet 27095a9813314d12804fa01ff5532b3c (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.626204 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27095a9813314d12804fa01ff5532b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.628086 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 27095a9813314d12804fa01ff5532b3c (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.628574 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27095a9813314d12804fa01ff5532b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.630002 8106 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.631757 8106 tablet_bootstrap.cc:654] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.638661 7379 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Sending DeleteTablet for 3 replicas of tablet 05406776807c49419ee4067df0b023a9
I20240503 02:13:11.637625 7971 tablet_service.cc:1467] Processing CreateTablet for tablet 27095a9813314d12804fa01ff5532b3c (DEFAULT_TABLE table=default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.639344 7379 catalog_manager.cc:6554] ReplaceTablet: tablet 05406776807c49419ee4067df0b023a9 deleted and replaced by tablet fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:11.639636 7379 master_service.cc:946] ReplaceTablet: received request to replace tablet 2906301538e04dfb962fff2381023ba0 from {username='slave'} at 127.0.0.1:36498
I20240503 02:13:11.639770 7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27095a9813314d12804fa01ff5532b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.640277 7379 master_service.cc:946] ReplaceTablet: received request to replace tablet 05406776807c49419ee4067df0b023a9 from {username='slave'} at 127.0.0.1:36498
I20240503 02:13:11.642305 7832 tablet_service.cc:1514] Processing DeleteTablet for tablet 05406776807c49419ee4067df0b023a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fb1acfaf1d3d4673b89f0de854e60d0e) from {username='slave'} at 127.0.0.1:40404
I20240503 02:13:11.644945 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dca752b6947f8a891a9af6a58fdf4" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:11.645490 7712 raft_consensus.cc:2454] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:11.646291 7790 leader_election.cc:302] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.646641 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet 05406776807c49419ee4067df0b023a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fb1acfaf1d3d4673b89f0de854e60d0e) from {username='slave'} at 127.0.0.1:33542
I20240503 02:13:11.647007 8115 raft_consensus.cc:2781] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.647137 8115 raft_consensus.cc:482] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.647230 8115 raft_consensus.cc:3037] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.647478 7382 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_bcc90e6508b24abfafc68d6715194f7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:11.648281 7379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_721855476892497ba8ae9b4ad7930e99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:11.650158 8115 raft_consensus.cc:504] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.651801 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dca752b6947f8a891a9af6a58fdf4" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:11.652026 7712 raft_consensus.cc:3037] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.655117 7712 raft_consensus.cc:2454] T 8a8dca752b6947f8a891a9af6a58fdf4 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:11.655791 7790 leader_election.cc:302] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.658321 8115 leader_election.cc:288] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:11.659016 8115 raft_consensus.cc:2781] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.659512 8115 raft_consensus.cc:686] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 LEADER]: Becoming Leader. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Running, Role: LEADER
I20240503 02:13:11.659459 7689 tablet_service.cc:1467] Processing CreateTablet for tablet fb1acfaf1d3d4673b89f0de854e60d0e (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.660111 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1acfaf1d3d4673b89f0de854e60d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.660807 8115 consensus_queue.cc:237] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.661186 8109 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.663448 8109 tablet_bootstrap.cc:654] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.665692 7969 tablet_service.cc:1514] Processing DeleteTablet for tablet 05406776807c49419ee4067df0b023a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fb1acfaf1d3d4673b89f0de854e60d0e) from {username='slave'} at 127.0.0.1:58384
I20240503 02:13:11.668161 8126 ts_tablet_manager.cc:1875] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.669720 8125 tablet_replica.cc:330] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: stopping tablet replica
I20240503 02:13:11.669924 7833 tablet_service.cc:1467] Processing CreateTablet for tablet fb1acfaf1d3d4673b89f0de854e60d0e (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.672082 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1acfaf1d3d4673b89f0de854e60d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.672152 8125 raft_consensus.cc:2229] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.674024 8125 raft_consensus.cc:2258] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.679121 8106 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.679311 8106 ts_tablet_manager.cc:1367] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.003s
I20240503 02:13:11.680202 8106 raft_consensus.cc:348] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.680431 8106 raft_consensus.cc:374] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.680545 8106 raft_consensus.cc:729] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.680835 8106 consensus_queue.cc:260] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.681435 8106 ts_tablet_manager.cc:1398] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:11.681761 8106 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.683547 8106 tablet_bootstrap.cc:654] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.685513 7971 tablet_service.cc:1467] Processing CreateTablet for tablet fb1acfaf1d3d4673b89f0de854e60d0e (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.685982 7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1acfaf1d3d4673b89f0de854e60d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.690009 8126 ts_tablet_manager.cc:1888] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:13:11.690529 8126 log.cc:1196] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/2906301538e04dfb962fff2381023ba0
I20240503 02:13:11.691568 8126 ts_tablet_manager.cc:1909] T 2906301538e04dfb962fff2381023ba0 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting consensus metadata
I20240503 02:13:11.675877 8124 tablet_replica.cc:330] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: stopping tablet replica
I20240503 02:13:11.685792 8108 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.692214 8108 ts_tablet_manager.cc:1367] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.010s
I20240503 02:13:11.692351 8106 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.692523 8106 ts_tablet_manager.cc:1367] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240503 02:13:11.692651 8124 raft_consensus.cc:2229] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:11.693353 8124 pending_rounds.cc:62] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Trying to abort 1 pending ops.
I20240503 02:13:11.693331 8106 raft_consensus.cc:348] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.693364 8108 raft_consensus.cc:348] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.693567 8106 raft_consensus.cc:374] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.693526 8124 pending_rounds.cc:68] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023420995869417472 op_type: NO_OP noop_request { }
I20240503 02:13:11.693660 8108 raft_consensus.cc:374] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.693670 8106 raft_consensus.cc:729] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.693751 8124 raft_consensus.cc:2866] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:13:11.693871 8124 raft_consensus.cc:2258] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.693989 8108 raft_consensus.cc:729] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.693979 8106 consensus_queue.cc:260] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.694234 8109 tablet_bootstrap.cc:492] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.694427 8109 ts_tablet_manager.cc:1367] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20240503 02:13:11.694959 8108 consensus_queue.cc:260] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.695526 8109 raft_consensus.cc:348] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.695741 8109 raft_consensus.cc:374] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.695838 8109 raft_consensus.cc:729] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.696071 8109 consensus_queue.cc:260] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.696646 8109 ts_tablet_manager.cc:1398] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:11.697522 8109 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.698987 8108 ts_tablet_manager.cc:1398] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:11.699770 8109 tablet_bootstrap.cc:654] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.699801 8108 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.702287 7381 catalog_manager.cc:2145] created HMS catalog entry for table [id=ba32e3b5c4bf4aa8874f5afa875ccf52]
I20240503 02:13:11.702399 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dca752b6947f8a891a9af6a58fdf4" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:11.702667 7990 raft_consensus.cc:2454] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:11.702996 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dca752b6947f8a891a9af6a58fdf4" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:11.703162 7991 raft_consensus.cc:3037] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.705513 8125 ts_tablet_manager.cc:1875] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.705994 7991 raft_consensus.cc:2454] T 8a8dca752b6947f8a891a9af6a58fdf4 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:11.710476 8106 ts_tablet_manager.cc:1398] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:13:11.711992 8124 ts_tablet_manager.cc:1875] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:11.718298 7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.17.65:45901, user_credentials={real_user=slave}} blocked reactor thread for 62512.3us
I20240503 02:13:11.722088 8106 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.722937 7833 tablet_service.cc:1467] Processing CreateTablet for tablet d818c02e5f8c4bc8816a5cc702cf0b78 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.723381 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818c02e5f8c4bc8816a5cc702cf0b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.723814 8106 tablet_bootstrap.cc:654] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.727429 7691 tablet_service.cc:1467] Processing CreateTablet for tablet d818c02e5f8c4bc8816a5cc702cf0b78 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.729046 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818c02e5f8c4bc8816a5cc702cf0b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.731194 7969 tablet_service.cc:1467] Processing CreateTablet for tablet d818c02e5f8c4bc8816a5cc702cf0b78 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.731657 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818c02e5f8c4bc8816a5cc702cf0b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.736109 8114 raft_consensus.cc:482] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.736336 8114 raft_consensus.cc:504] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.736773 7377 catalog_manager.cc:2145] created HMS catalog entry for table [id=45b26215e3024493a872c4c7e0261d7e]
I20240503 02:13:11.737140 8114 leader_election.cc:288] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:11.737805 8112 raft_consensus.cc:482] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.737867 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7099ff89ceb544119b317319a4d0aa23" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:11.738116 7855 raft_consensus.cc:2454] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:11.737994 8112 raft_consensus.cc:504] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.738660 7645 leader_election.cc:302] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.740146 8112 leader_election.cc:288] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:11.741974 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7099ff89ceb544119b317319a4d0aa23" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:11.742228 7991 raft_consensus.cc:2454] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:11.744455 8114 raft_consensus.cc:2781] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.744585 8114 raft_consensus.cc:482] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.744701 8114 raft_consensus.cc:3037] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.737341 7381 catalog_manager.cc:5484] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66). New cstate: current_term: 1 leader_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.745752 7377 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_47870538eb6041179084ffe5cf79566b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:11.747614 8108 tablet_bootstrap.cc:654] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.747490 8114 raft_consensus.cc:504] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.748746 8125 ts_tablet_manager.cc:1888] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:13:11.748816 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7099ff89ceb544119b317319a4d0aa23" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:11.749109 7855 raft_consensus.cc:3037] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.751281 8124 ts_tablet_manager.cc:1888] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:13:11.751504 8124 log.cc:1196] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/2906301538e04dfb962fff2381023ba0
I20240503 02:13:11.751844 7855 raft_consensus.cc:2454] T 7099ff89ceb544119b317319a4d0aa23 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:11.752038 8124 ts_tablet_manager.cc:1909] T 2906301538e04dfb962fff2381023ba0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting consensus metadata
I20240503 02:13:11.752368 7645 leader_election.cc:302] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.752692 8122 raft_consensus.cc:2781] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.765331 8106 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.765532 8106 ts_tablet_manager.cc:1367] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.002s
I20240503 02:13:11.765767 8126 tablet_replica.cc:330] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: stopping tablet replica
I20240503 02:13:11.765857 7373 catalog_manager.cc:4830] TS c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): tablet 2906301538e04dfb962fff2381023ba0 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]) successfully deleted
I20240503 02:13:11.765992 8126 raft_consensus.cc:2229] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.766162 8126 raft_consensus.cc:2258] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.766088 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05406776807c49419ee4067df0b023a9" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:11.766327 7855 raft_consensus.cc:2454] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:11.766371 8106 raft_consensus.cc:348] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.766563 8106 raft_consensus.cc:374] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.766652 8106 raft_consensus.cc:729] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.766871 8106 consensus_queue.cc:260] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.767171 7924 leader_election.cc:302] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.767479 8106 ts_tablet_manager.cc:1398] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 02:13:11.767537 7924 raft_consensus.cc:2670] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 02:13:11.767814 8106 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.768008 8126 ts_tablet_manager.cc:1875] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.769599 8106 tablet_bootstrap.cc:654] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.774039 8108 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.774272 8108 ts_tablet_manager.cc:1367] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.004s
I20240503 02:13:11.775333 8108 raft_consensus.cc:348] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.775628 8108 raft_consensus.cc:374] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.775708 8108 raft_consensus.cc:729] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.776053 8108 consensus_queue.cc:260] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.778605 8109 tablet_bootstrap.cc:492] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.778684 8106 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.778815 8109 ts_tablet_manager.cc:1367] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.001s
I20240503 02:13:11.778867 8106 ts_tablet_manager.cc:1367] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:13:11.779702 8109 raft_consensus.cc:348] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.779974 8109 raft_consensus.cc:374] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.779889 8106 raft_consensus.cc:348] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.780045 8109 raft_consensus.cc:729] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.780124 8106 raft_consensus.cc:374] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.780200 8106 raft_consensus.cc:729] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.780485 8106 consensus_queue.cc:260] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.780772 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9a18032b2f4b48a9b99910fa28f16f (DEFAULT_TABLE table=default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.780954 8122 raft_consensus.cc:686] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:11.781244 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9a18032b2f4b48a9b99910fa28f16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.781307 8122 consensus_queue.cc:237] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.782359 8106 ts_tablet_manager.cc:1398] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:11.782910 7381 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36498:
name: "default.Table_a15ad42e445d4011a7ce7a358eb31e66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:11.783952 7969 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9a18032b2f4b48a9b99910fa28f16f (DEFAULT_TABLE table=default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.784400 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9a18032b2f4b48a9b99910fa28f16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.785130 8109 consensus_queue.cc:260] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.785727 8109 ts_tablet_manager.cc:1398] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:13:11.786058 8109 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.787794 8109 tablet_bootstrap.cc:654] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.790621 8126 ts_tablet_manager.cc:1888] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:13:11.790808 8126 log.cc:1196] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/05406776807c49419ee4067df0b023a9
I20240503 02:13:11.795413 7372 catalog_manager.cc:4830] TS 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): tablet 2906301538e04dfb962fff2381023ba0 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]) successfully deleted
W20240503 02:13:11.795892 7372 catalog_manager.cc:4797] TS 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): delete failed for tablet 2906301538e04dfb962fff2381023ba0 because the tablet was not found. No further retry: Not found: Tablet not found: 2906301538e04dfb962fff2381023ba0
I20240503 02:13:11.797003 7691 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9a18032b2f4b48a9b99910fa28f16f (DEFAULT_TABLE table=default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.799044 8126 ts_tablet_manager.cc:1909] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting consensus metadata
I20240503 02:13:11.799968 8106 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:11.799990 7373 catalog_manager.cc:4830] TS c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): tablet 05406776807c49419ee4067df0b023a9 (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]) successfully deleted
I20240503 02:13:11.801750 8106 tablet_bootstrap.cc:654] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.807725 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:34141
--webserver_interface=127.4.17.124
--webserver_port=34607
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:13:11.810223 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9a18032b2f4b48a9b99910fa28f16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.817314 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7099ff89ceb544119b317319a4d0aa23" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:11.817704 7991 raft_consensus.cc:3037] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.821039 8114 leader_election.cc:288] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Requested vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:11.821930 7991 raft_consensus.cc:2454] T 7099ff89ceb544119b317319a4d0aa23 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:11.826500 8109 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.826711 8124 tablet_replica.cc:330] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: stopping tablet replica
I20240503 02:13:11.826720 8109 ts_tablet_manager.cc:1367] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20240503 02:13:11.826949 8124 raft_consensus.cc:2229] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.827199 8124 raft_consensus.cc:2258] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.827461 7386 catalog_manager.cc:2145] created HMS catalog entry for table [id=9a298c9ff14e4f6babe912e4148e4334]
I20240503 02:13:11.828174 8124 ts_tablet_manager.cc:1875] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.748941 8125 log.cc:1196] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/2906301538e04dfb962fff2381023ba0
I20240503 02:13:11.834416 8125 ts_tablet_manager.cc:1909] T 2906301538e04dfb962fff2381023ba0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting consensus metadata
I20240503 02:13:11.838949 8109 raft_consensus.cc:348] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.839218 8109 raft_consensus.cc:374] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.839416 8109 raft_consensus.cc:729] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.839715 8109 consensus_queue.cc:260] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.839160 8124 ts_tablet_manager.cc:1888] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:13:11.841073 8124 log.cc:1196] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-0/wal/wals/05406776807c49419ee4067df0b023a9
I20240503 02:13:11.841818 8124 ts_tablet_manager.cc:1909] T 05406776807c49419ee4067df0b023a9 P 6f6a6a27cc814e0a86b1767df5f5a31b: Deleting consensus metadata
I20240503 02:13:11.844071 7372 catalog_manager.cc:4830] TS 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): tablet 05406776807c49419ee4067df0b023a9 (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]) successfully deleted
I20240503 02:13:11.845999 7386 catalog_manager.cc:5484] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65). New cstate: current_term: 1 leader_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.846454 7691 tablet_service.cc:1467] Processing CreateTablet for tablet e8d80f4253024847b86b0fdc6f31e175 (DEFAULT_TABLE table=default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.846902 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d80f4253024847b86b0fdc6f31e175. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:11.846930 7926 leader_election.cc:341] T 05406776807c49419ee4067df0b023a9 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 05406776807c49419ee4067df0b023a9
I20240503 02:13:11.843071 8109 ts_tablet_manager.cc:1398] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20240503 02:13:11.849879 7833 tablet_service.cc:1467] Processing CreateTablet for tablet e8d80f4253024847b86b0fdc6f31e175 (DEFAULT_TABLE table=default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.850322 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d80f4253024847b86b0fdc6f31e175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.854806 7969 tablet_service.cc:1467] Processing CreateTablet for tablet e8d80f4253024847b86b0fdc6f31e175 (DEFAULT_TABLE table=default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.855333 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d80f4253024847b86b0fdc6f31e175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.865226 7386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_bd10243fcd8e4436952edc7daf12e223" } new_table_name: "default.Table_ffe0555a64a74d05b9816a5be84d179c" modify_external_catalogs: true
I20240503 02:13:11.866844 8125 tablet_replica.cc:330] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: stopping tablet replica
I20240503 02:13:11.866870 7370 catalog_manager.cc:4830] TS c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): tablet 2906301538e04dfb962fff2381023ba0 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]) successfully deleted
I20240503 02:13:11.868340 8125 raft_consensus.cc:2229] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.868423 8108 ts_tablet_manager.cc:1398] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.094s user 0.021s sys 0.025s
I20240503 02:13:11.869021 8108 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.870940 8108 tablet_bootstrap.cc:654] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.850332 8109 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.868574 8125 raft_consensus.cc:2258] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.878012 8106 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.878058 8109 tablet_bootstrap.cc:654] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.878206 8106 ts_tablet_manager.cc:1367] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.015s
I20240503 02:13:11.878648 8125 ts_tablet_manager.cc:1875] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.879105 8106 raft_consensus.cc:348] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.879408 8106 raft_consensus.cc:374] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.879525 8106 raft_consensus.cc:729] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.879952 8106 consensus_queue.cc:260] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.883805 8106 ts_tablet_manager.cc:1398] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:11.888147 8125 ts_tablet_manager.cc:1888] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:13:11.891296 8112 raft_consensus.cc:482] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.891446 8125 log.cc:1196] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-1/wal/wals/05406776807c49419ee4067df0b023a9
I20240503 02:13:11.891744 8112 raft_consensus.cc:504] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.892570 8125 ts_tablet_manager.cc:1909] T 05406776807c49419ee4067df0b023a9 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Deleting consensus metadata
I20240503 02:13:11.892777 8112 leader_election.cc:288] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:11.893806 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:11.894495 7370 catalog_manager.cc:4830] TS c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): tablet 05406776807c49419ee4067df0b023a9 (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]) successfully deleted
W20240503 02:13:11.895363 7924 leader_election.cc:341] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:11.895484 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:11.896250 8106 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
W20240503 02:13:11.896976 7926 leader_election.cc:341] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:11.897130 7926 leader_election.cc:302] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:11.897742 8112 raft_consensus.cc:2726] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:11.898289 8106 tablet_bootstrap.cc:654] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.910066 8106 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:11.910311 8106 ts_tablet_manager.cc:1367] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:13:11.911415 8106 raft_consensus.cc:348] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.911689 8106 raft_consensus.cc:374] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.911769 8106 raft_consensus.cc:729] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.912171 8106 consensus_queue.cc:260] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.915000 8108 tablet_bootstrap.cc:492] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.915053 8106 ts_tablet_manager.cc:1398] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:13:11.915230 8108 ts_tablet_manager.cc:1367] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.019s
I20240503 02:13:11.916321 8108 raft_consensus.cc:348] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.916574 8108 raft_consensus.cc:374] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.916675 8108 raft_consensus.cc:729] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.917001 8108 consensus_queue.cc:260] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.918956 8108 ts_tablet_manager.cc:1398] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:11.919380 8108 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.921525 8108 tablet_bootstrap.cc:654] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.926765 8109 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.927085 8109 ts_tablet_manager.cc:1367] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.019s
I20240503 02:13:11.928081 8109 raft_consensus.cc:348] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.928416 8109 raft_consensus.cc:374] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.928547 8109 raft_consensus.cc:729] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.928948 8109 consensus_queue.cc:260] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.934197 8109 ts_tablet_manager.cc:1398] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:11.937722 8109 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.938438 8108 tablet_bootstrap.cc:492] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.938735 8108 ts_tablet_manager.cc:1367] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20240503 02:13:11.939878 8109 tablet_bootstrap.cc:654] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.941272 8108 raft_consensus.cc:348] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.941613 8108 raft_consensus.cc:374] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.941713 8108 raft_consensus.cc:729] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.942140 8108 consensus_queue.cc:260] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.946378 8108 ts_tablet_manager.cc:1398] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:11.947602 8108 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.949779 8108 tablet_bootstrap.cc:654] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.950785 8112 raft_consensus.cc:482] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.951002 8112 raft_consensus.cc:504] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.951180 8109 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.951350 8109 ts_tablet_manager.cc:1367] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240503 02:13:11.951864 8112 leader_election.cc:288] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:11.952277 8109 raft_consensus.cc:348] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.952492 8109 raft_consensus.cc:374] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.952625 8109 raft_consensus.cc:729] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.952875 8109 consensus_queue.cc:260] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.953029 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:11.953253 7712 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:11.953446 8109 ts_tablet_manager.cc:1398] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:11.953645 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:11.953809 8109 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:11.953876 7855 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:11.955740 8109 tablet_bootstrap.cc:654] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.957067 7924 leader_election.cc:302] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:11.960512 8112 raft_consensus.cc:2781] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.960700 8112 raft_consensus.cc:482] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.960829 8112 raft_consensus.cc:3037] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.966241 8112 raft_consensus.cc:504] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.968254 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:11.968617 7712 raft_consensus.cc:3037] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.969333 8112 leader_election.cc:288] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Requested vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:11.970247 8109 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:11.970482 8109 ts_tablet_manager.cc:1367] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20240503 02:13:11.971278 8109 raft_consensus.cc:348] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.971557 8109 raft_consensus.cc:374] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.971566 7712 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:11.971645 8109 raft_consensus.cc:729] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.971930 8109 consensus_queue.cc:260] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.972473 8109 ts_tablet_manager.cc:1398] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:11.972970 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:11.973197 7855 raft_consensus.cc:3037] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.975996 7855 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:11.976943 7926 leader_election.cc:302] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:11.977348 7384 catalog_manager.cc:2145] created HMS catalog entry for table [id=5069a59fcea74f718a1b945cc20aab7f]
I20240503 02:13:11.978165 8108 tablet_bootstrap.cc:492] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:11.978405 8108 ts_tablet_manager.cc:1367] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.002s
I20240503 02:13:11.979317 8108 raft_consensus.cc:348] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.979542 8108 raft_consensus.cc:374] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.979678 8108 raft_consensus.cc:729] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.979947 8108 consensus_queue.cc:260] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:11.980504 8108 ts_tablet_manager.cc:1398] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:11.980806 8108 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:11.985077 7384 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36498 (request call id 2) took 1146 ms. Trace:
I20240503 02:13:11.985248 7384 rpcz_store.cc:276] 0503 02:13:10.839049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:13:10.841907 (+ 2858us) service_pool.cc:225] Handling call
0503 02:13:11.152849 (+310942us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:13:11.153275 (+ 426us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:13:11.977314 (+824039us) catalog_manager.cc:2144] Created new table in HMS catalog
0503 02:13:11.984691 (+ 7377us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:13:11.984826 (+ 135us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:13:11.984983 (+ 157us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:13:11.978182 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:13:11.978315 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:13:11.978322 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:13:11.978324 (+ 2us) tablet.cc:660] Decoding operations
0503 02:13:11.978443 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:13:11.978460 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0503 02:13:11.978468 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:13:11.978563 (+ 95us) tablet.cc:699] Row locks acquired
0503 02:13:11.978566 (+ 3us) write_op.cc:260] PREPARE: finished
0503 02:13:11.978614 (+ 48us) write_op.cc:270] Start()
0503 02:13:11.978657 (+ 43us) write_op.cc:276] Timestamp: P: 1714702391978591 usec, L: 0
0503 02:13:11.978661 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:13:11.978888 (+ 227us) log.cc:842] Serialized 432 byte log entry
0503 02:13:11.983916 (+ 5028us) op_driver.cc:444] REPLICATION: finished
0503 02:13:11.984069 (+ 153us) write_op.cc:301] APPLY: starting
0503 02:13:11.984091 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 02:13:11.984144 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0503 02:13:11.984184 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:13:11.984316 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:13:11.984359 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:13:11.984368 (+ 9us) write_op.cc:312] APPLY: finished
0503 02:13:11.984466 (+ 98us) log.cc:842] Serialized 28 byte log entry
0503 02:13:11.984544 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:13:11.984574 (+ 30us) write_op.cc:454] Released schema lock
0503 02:13:11.984582 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:13:11.984659 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:13:11.984663 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":673741,"Hive Metastore.run_cpu_time_us":316,"Hive Metastore.run_wall_time_us":150075,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":655,"apply.run_wall_time_us":654,"num_ops":2,"prepare.queue_time_us":138,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":827,"raft.queue_time_us":157,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":293,"replication_time_us":5223}]]}
I20240503 02:13:11.986847 8108 tablet_bootstrap.cc:654] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.001557 7833 tablet_service.cc:1467] Processing CreateTablet for tablet be53b36465e2461dbea23d101f957318 (DEFAULT_TABLE table=default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.001858 7969 tablet_service.cc:1467] Processing CreateTablet for tablet be53b36465e2461dbea23d101f957318 (DEFAULT_TABLE table=default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.002099 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be53b36465e2461dbea23d101f957318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.002333 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be53b36465e2461dbea23d101f957318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.004153 7384 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_82fc16f07bdb4d24bb75d0b04ffe9c29" } new_table_name: "default.Table_ec90921d196d413d8dc49d7ed15da51e" modify_external_catalogs: true
I20240503 02:13:12.007496 7691 tablet_service.cc:1467] Processing CreateTablet for tablet be53b36465e2461dbea23d101f957318 (DEFAULT_TABLE table=default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.008011 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be53b36465e2461dbea23d101f957318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.009040 8109 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:12.011525 8109 tablet_bootstrap.cc:654] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.021404 8112 raft_consensus.cc:2781] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.027369 8109 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:12.027601 8109 ts_tablet_manager.cc:1367] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240503 02:13:12.029048 8109 raft_consensus.cc:348] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.029345 8109 raft_consensus.cc:374] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.029469 8109 raft_consensus.cc:729] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.029840 8109 consensus_queue.cc:260] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.032102 8109 ts_tablet_manager.cc:1398] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:13:12.039805 8112 raft_consensus.cc:686] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 LEADER]: Becoming Leader. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Running, Role: LEADER
I20240503 02:13:12.040371 8112 consensus_queue.cc:237] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.043522 8106 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:12.045397 8106 tablet_bootstrap.cc:654] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.079069 8106 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:12.079277 8106 ts_tablet_manager.cc:1367] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.013s
I20240503 02:13:12.080492 7383 catalog_manager.cc:2145] created HMS catalog entry for table [id=bb7ee1fde2f145feaf1ded54deff266c]
I20240503 02:13:12.082094 8108 tablet_bootstrap.cc:492] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:12.082357 8108 ts_tablet_manager.cc:1367] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.102s user 0.030s sys 0.038s
I20240503 02:13:12.083214 8108 raft_consensus.cc:348] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.083521 8108 raft_consensus.cc:374] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.083650 8108 raft_consensus.cc:729] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.083953 8108 consensus_queue.cc:260] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.084554 8108 ts_tablet_manager.cc:1398] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:12.084939 8108 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:12.086787 8108 tablet_bootstrap.cc:654] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.087495 7383 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36498 (request call id 5) took 1248 ms. Trace:
I20240503 02:13:12.087620 7383 rpcz_store.cc:276] 0503 02:13:10.839457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:13:10.839566 (+ 109us) service_pool.cc:225] Handling call
0503 02:13:11.153816 (+314250us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 02:13:11.154281 (+ 465us) catalog_manager.cc:2118] Created new table and tablet info
0503 02:13:12.080446 (+926165us) catalog_manager.cc:2144] Created new table in HMS catalog
0503 02:13:12.087170 (+ 6724us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 02:13:12.087303 (+ 133us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 02:13:12.087430 (+ 127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:13:12.081686 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 02:13:12.081815 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:13:12.081821 (+ 6us) write_op.cc:435] Acquired schema lock
0503 02:13:12.081823 (+ 2us) tablet.cc:660] Decoding operations
0503 02:13:12.081931 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:13:12.081950 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0503 02:13:12.081955 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 02:13:12.082038 (+ 83us) tablet.cc:699] Row locks acquired
0503 02:13:12.082040 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:13:12.082085 (+ 45us) write_op.cc:270] Start()
0503 02:13:12.082121 (+ 36us) write_op.cc:276] Timestamp: P: 1714702392082053 usec, L: 0
0503 02:13:12.082125 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 02:13:12.082338 (+ 213us) log.cc:842] Serialized 432 byte log entry
0503 02:13:12.086454 (+ 4116us) op_driver.cc:444] REPLICATION: finished
0503 02:13:12.086595 (+ 141us) write_op.cc:301] APPLY: starting
0503 02:13:12.086615 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 02:13:12.086666 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0503 02:13:12.086707 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:13:12.086822 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:13:12.086862 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:13:12.086870 (+ 8us) write_op.cc:312] APPLY: finished
0503 02:13:12.086961 (+ 91us) log.cc:842] Serialized 28 byte log entry
0503 02:13:12.087033 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:13:12.087057 (+ 24us) write_op.cc:454] Released schema lock
0503 02:13:12.087065 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 02:13:12.087139 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:13:12.087143 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":823172,"Hive Metastore.run_cpu_time_us":338,"Hive Metastore.run_wall_time_us":102644,"spinlock_wait_cycles":115968,"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":603,"apply.run_wall_time_us":602,"num_ops":2,"prepare.queue_time_us":377,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":781,"raft.queue_time_us":902,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":282,"replication_time_us":4301}]]}
I20240503 02:13:12.101492 7969 tablet_service.cc:1467] Processing CreateTablet for tablet f3014429e2a54d01ba806f9b26e58485 (DEFAULT_TABLE table=default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.101999 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3014429e2a54d01ba806f9b26e58485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.106760 7833 tablet_service.cc:1467] Processing CreateTablet for tablet f3014429e2a54d01ba806f9b26e58485 (DEFAULT_TABLE table=default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.107210 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3014429e2a54d01ba806f9b26e58485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.107903 7691 tablet_service.cc:1467] Processing CreateTablet for tablet f3014429e2a54d01ba806f9b26e58485 (DEFAULT_TABLE table=default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.108338 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3014429e2a54d01ba806f9b26e58485. 1 dirs total, 0 dirs full, 0 dirs failed
F20240503 02:13:12.111306 7624 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cba0c48d8df441209b09207d54435b46
*** Check failure stack trace: ***
@ 0x7f7777c9fdcd google::LogMessage::Fail() at ??:0
I20240503 02:13:12.116106 8115 raft_consensus.cc:482] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.116250 8108 tablet_bootstrap.cc:492] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:12.116410 8108 ts_tablet_manager.cc:1367] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.003s
I20240503 02:13:12.116310 8115 raft_consensus.cc:504] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.117103 8115 leader_election.cc:288] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:12.117204 8108 raft_consensus.cc:348] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.117420 8108 raft_consensus.cc:374] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.117915 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d80f4253024847b86b0fdc6f31e175" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:12.117918 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d80f4253024847b86b0fdc6f31e175" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:12.118139 7712 raft_consensus.cc:2454] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:12.118131 7991 raft_consensus.cc:2454] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:12.118609 7790 leader_election.cc:302] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.119146 8115 raft_consensus.cc:2781] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.119256 8115 raft_consensus.cc:482] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.119357 8115 raft_consensus.cc:3037] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.121980 8115 raft_consensus.cc:504] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.122730 8106 raft_consensus.cc:348] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.122959 8106 raft_consensus.cc:374] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.123049 8106 raft_consensus.cc:729] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.123301 8106 consensus_queue.cc:260] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.123400 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d80f4253024847b86b0fdc6f31e175" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:12.123638 7712 raft_consensus.cc:3037] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.123926 8106 ts_tablet_manager.cc:1398] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20240503 02:13:12.124238 8106 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:12.126444 7712 raft_consensus.cc:2454] T e8d80f4253024847b86b0fdc6f31e175 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:12.127261 7790 leader_election.cc:302] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.127610 8130 raft_consensus.cc:2781] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.130074 8115 leader_election.cc:288] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:12.130154 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d80f4253024847b86b0fdc6f31e175" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:12.130431 7991 raft_consensus.cc:3037] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.131320 8106 tablet_bootstrap.cc:654] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.134788 8109 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:12.136577 8109 tablet_bootstrap.cc:654] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.136890 7991 raft_consensus.cc:2454] T e8d80f4253024847b86b0fdc6f31e175 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:12.142685 8130 raft_consensus.cc:686] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 LEADER]: Becoming Leader. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Running, Role: LEADER
I20240503 02:13:12.143247 8130 consensus_queue.cc:237] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.117516 8108 raft_consensus.cc:729] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
@ 0x7f7777ca3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7777c9f7cc google::LogMessage::Flush() at ??:0
@ 0x7f7777c9fa11 google::LogMessage::~LogMessage() at ??:0
I20240503 02:13:12.153277 7383 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
I20240503 02:13:12.153556 7383 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36498:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
I20240503 02:13:12.145061 8108 consensus_queue.cc:260] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.157017 8106 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:12.157582 8106 ts_tablet_manager.cc:1367] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.016s
I20240503 02:13:12.158390 8106 raft_consensus.cc:348] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.158643 8106 raft_consensus.cc:374] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.158708 8106 raft_consensus.cc:729] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.158983 8106 consensus_queue.cc:260] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.158922 8108 ts_tablet_manager.cc:1398] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
I20240503 02:13:12.159555 8106 ts_tablet_manager.cc:1398] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:12.159848 8108 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:12.161743 8108 tablet_bootstrap.cc:654] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
@ 0x7f778deb557e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240503 02:13:12.175025 8108 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:12.175274 8108 ts_tablet_manager.cc:1367] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
@ 0x7f778deb4261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240503 02:13:12.176292 8108 raft_consensus.cc:348] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.176609 8108 raft_consensus.cc:374] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.176689 8108 raft_consensus.cc:729] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.176998 8108 consensus_queue.cc:260] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.179158 8109 tablet_bootstrap.cc:492] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:12.179344 8109 ts_tablet_manager.cc:1367] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.002s
I20240503 02:13:12.180364 8109 raft_consensus.cc:348] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.180806 8109 raft_consensus.cc:374] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.180878 8109 raft_consensus.cc:729] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.181202 8109 consensus_queue.cc:260] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
@ 0x7f778debb2bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20240503 02:13:12.182132 8109 ts_tablet_manager.cc:1398] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:13:12.188997 8108 ts_tablet_manager.cc:1398] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
@ 0x7f778debb139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f777a98f01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f777dd146db start_thread at ??:0
@ 0x7f7776b5271f clone at ??:0
I20240503 02:13:12.293525 8141 raft_consensus.cc:482] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.293745 8141 raft_consensus.cc:504] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.294531 8141 leader_election.cc:288] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.295203 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be53b36465e2461dbea23d101f957318" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:12.295311 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be53b36465e2461dbea23d101f957318" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:12.295507 7991 raft_consensus.cc:2454] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:12.295562 7855 raft_consensus.cc:2454] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:12.296010 7648 leader_election.cc:302] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:12.296281 8141 raft_consensus.cc:2781] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.296394 8141 raft_consensus.cc:482] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.296499 8141 raft_consensus.cc:3037] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.298516 8141 raft_consensus.cc:504] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.299320 8141 leader_election.cc:288] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Requested vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.299759 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be53b36465e2461dbea23d101f957318" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:12.299770 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be53b36465e2461dbea23d101f957318" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:12.300019 7991 raft_consensus.cc:3037] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.300025 7855 raft_consensus.cc:3037] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.303081 7991 raft_consensus.cc:2454] T be53b36465e2461dbea23d101f957318 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:12.303089 7855 raft_consensus.cc:2454] T be53b36465e2461dbea23d101f957318 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:12.303742 7645 leader_election.cc:302] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.304236 8141 raft_consensus.cc:2781] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.304778 8141 raft_consensus.cc:686] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:12.305127 8141 consensus_queue.cc:237] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:12.320904 8112 raft_consensus.cc:482] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.321192 8112 raft_consensus.cc:504] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.321946 8112 leader_election.cc:288] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.322559 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27095a9813314d12804fa01ff5532b3c" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:12.322589 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27095a9813314d12804fa01ff5532b3c" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:12.322835 7855 raft_consensus.cc:2454] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:12.322844 7712 raft_consensus.cc:2454] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:12.323431 7926 leader_election.cc:302] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:12.323776 8112 raft_consensus.cc:2781] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.323906 8112 raft_consensus.cc:482] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.324000 8112 raft_consensus.cc:3037] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.327059 8112 raft_consensus.cc:504] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.327867 8112 leader_election.cc:288] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Requested vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.328737 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27095a9813314d12804fa01ff5532b3c" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:12.328723 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27095a9813314d12804fa01ff5532b3c" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:12.328995 7712 raft_consensus.cc:3037] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.329054 7855 raft_consensus.cc:3037] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.332036 7712 raft_consensus.cc:2454] T 27095a9813314d12804fa01ff5532b3c P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:12.332644 7855 raft_consensus.cc:2454] T 27095a9813314d12804fa01ff5532b3c P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:12.332698 7926 leader_election.cc:302] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:12.333216 8112 raft_consensus.cc:2781] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.334031 8112 raft_consensus.cc:686] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 LEADER]: Becoming Leader. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Running, Role: LEADER
I20240503 02:13:12.334380 8112 consensus_queue.cc:237] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.358263 8141 raft_consensus.cc:482] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.358554 8141 raft_consensus.cc:504] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.360122 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:12.360406 7991 raft_consensus.cc:2454] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:12.360399 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:12.360987 7648 leader_election.cc:302] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:12.361320 8114 raft_consensus.cc:2781] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.361452 8114 raft_consensus.cc:482] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.361449 7855 raft_consensus.cc:2454] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:12.361546 8114 raft_consensus.cc:3037] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.362115 8141 leader_election.cc:288] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:12.364424 8114 raft_consensus.cc:504] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.365274 8114 leader_election.cc:288] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Requested vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:12.365754 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:12.366039 7855 raft_consensus.cc:3037] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.365983 7991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818c02e5f8c4bc8816a5cc702cf0b78" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:12.366204 7991 raft_consensus.cc:3037] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.369123 7991 raft_consensus.cc:2454] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:12.369742 7855 raft_consensus.cc:2454] T d818c02e5f8c4bc8816a5cc702cf0b78 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:12.369858 7648 leader_election.cc:302] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710; no voters:
I20240503 02:13:12.370290 8114 raft_consensus.cc:2781] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.370561 8114 raft_consensus.cc:686] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:12.370957 8114 consensus_queue.cc:237] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
W20240503 02:13:12.441546 8047 connection.cc:535] client connection to 127.4.17.126:42485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:12.442396 8047 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.126:42485)
W20240503 02:13:12.442765 8047 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.126:42485)
W20240503 02:13:12.443349 8047 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.126:42485)
W20240503 02:13:12.444343 8047 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.126:42485)
I20240503 02:13:12.466454 8112 raft_consensus.cc:482] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.466676 8112 raft_consensus.cc:504] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.467307 8112 leader_election.cc:288] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.467904 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9a18032b2f4b48a9b99910fa28f16f" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:12.467981 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9a18032b2f4b48a9b99910fa28f16f" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:12.468212 7712 raft_consensus.cc:2454] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:12.468205 7855 raft_consensus.cc:2454] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 0.
I20240503 02:13:12.468672 7924 leader_election.cc:302] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.469055 8112 raft_consensus.cc:2781] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.469166 8112 raft_consensus.cc:482] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.469288 8112 raft_consensus.cc:3037] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.471894 8112 raft_consensus.cc:504] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:12.472481 8112 leader_election.cc:288] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Requested vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.473029 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9a18032b2f4b48a9b99910fa28f16f" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:12.473052 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9a18032b2f4b48a9b99910fa28f16f" candidate_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:12.473280 7855 raft_consensus.cc:3037] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.473292 7712 raft_consensus.cc:3037] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.476155 7712 raft_consensus.cc:2454] T 0e9a18032b2f4b48a9b99910fa28f16f P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:12.476176 7855 raft_consensus.cc:2454] T 0e9a18032b2f4b48a9b99910fa28f16f P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67f5cc3ea7a4529a01ede7d54fe4710 in term 1.
I20240503 02:13:12.476661 7924 leader_election.cc:302] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.477028 8112 raft_consensus.cc:2781] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.477214 8112 raft_consensus.cc:686] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 LEADER]: Becoming Leader. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Running, Role: LEADER
I20240503 02:13:12.478868 8112 consensus_queue.cc:237] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
W20240503 02:13:12.502095 8147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:12.502588 8147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:12.502732 8147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:12.511541 8147 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:12.511672 8147 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:12.511724 8147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:12.511806 8147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:12.511873 8147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:12.511945 8147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:12.522679 8147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.124:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.17.124
--webserver_port=34607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:12.524207 8147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:12.526675 8147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:12.539769 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.541013 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.542202 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:12.543704 8147 server_base.cc:1034] running on GCE node
I20240503 02:13:12.544559 8147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:12.546247 8147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:12.547487 8147 hybrid_clock.cc:648] HybridClock initialized: now 1714702392547413 us; error 43 us; skew 500 ppm
I20240503 02:13:12.547878 8147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:12.556216 8147 webserver.cc:460] Webserver started at http://127.4.17.124:34607/ using document root <none> and password file <none>
I20240503 02:13:12.557086 8147 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:12.557233 8147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:12.563525 8147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:12.567029 8195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.568584 8147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:12.568776 8147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
format_stamp: "Formatted at 2024-05-03 02:13:08 on dist-test-slave-ddxx"
I20240503 02:13:12.569715 8147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:12.595758 8147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:12.596714 8147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:12.597019 8147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:12.599646 8202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f2855cc86e414615a261b7a7d4bd3293)
I20240503 02:13:12.599843 8202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:12.600586 8202 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:12.602857 7442 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:12.603078 7441 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:12.603195 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:12.603456 8202 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240503 02:13:12.613130 8114 consensus_queue.cc:1035] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.613395 8130 consensus_queue.cc:1035] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.630285 8147 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.124:34141
I20240503 02:13:12.630247 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.124:34141 every 8 connection(s)
I20240503 02:13:12.633515 8147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-2/data/info.pb
I20240503 02:13:12.640725 8160 consensus_queue.cc:1035] T 8a8dca752b6947f8a891a9af6a58fdf4 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.641036 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 8147
I20240503 02:13:12.652163 8141 consensus_queue.cc:1035] T 7099ff89ceb544119b317319a4d0aa23 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.652536 8255 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:12.672524 8255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Bootstrap starting.
I20240503 02:13:12.697252 8255 log.cc:824] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:12.702014 7900 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:12.702353 7900 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.704218 7757 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:12.704381 8036 heartbeater.cc:347] Connected to a master server at 127.4.17.124:34141
I20240503 02:13:12.704445 7757 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.704612 8036 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.707698 7757 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.707788 7900 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.709534 8036 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.712999 8255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:13:12.713613 8255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Bootstrap complete.
I20240503 02:13:12.720381 8255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:12.721583 8255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.722329 8255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:12.723217 8271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:12.723554 8271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:12.724370 8255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:12.740469 8285 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:12.740603 8285 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: loading cluster ID for follower catalog manager: success
I20240503 02:13:12.735069 8147 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111),
I20240503 02:13:12.743847 8147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:12.745350 8285 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: acquiring CA information for follower catalog manager: success
I20240503 02:13:12.750255 8285 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:12.962064 8256 raft_consensus.cc:482] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.962244 8130 consensus_queue.cc:1035] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.962293 8256 raft_consensus.cc:504] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.963002 8256 leader_election.cc:288] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:12.963447 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3014429e2a54d01ba806f9b26e58485" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:12.963646 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3014429e2a54d01ba806f9b26e58485" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
I20240503 02:13:12.963732 7990 raft_consensus.cc:2454] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:12.963912 7712 raft_consensus.cc:2454] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 0.
I20240503 02:13:12.964109 8036 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.964955 7791 leader_election.cc:302] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.965490 8256 raft_consensus.cc:2781] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.965639 8256 raft_consensus.cc:482] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.965728 8256 raft_consensus.cc:3037] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.968200 8256 raft_consensus.cc:504] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.969591 8256 leader_election.cc:288] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Requested vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:12.969926 8216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:49547; Asking this server to re-register.
I20240503 02:13:12.970312 7900 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.971253 8036 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.972486 7711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3014429e2a54d01ba806f9b26e58485" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b"
I20240503 02:13:12.972724 7711 raft_consensus.cc:3037] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.973811 8216 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:12.975416 7711 raft_consensus.cc:2454] T f3014429e2a54d01ba806f9b26e58485 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:12.975972 7790 leader_election.cc:302] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:12.976236 8216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:53547; Asking this server to re-register.
I20240503 02:13:12.976356 8160 consensus_queue.cc:1035] T e8d80f4253024847b86b0fdc6f31e175 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.976828 8160 raft_consensus.cc:2781] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.976984 8160 raft_consensus.cc:686] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 LEADER]: Becoming Leader. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Running, Role: LEADER
I20240503 02:13:12.977011 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3014429e2a54d01ba806f9b26e58485" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:12.977234 7990 raft_consensus.cc:3037] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.977289 8160 consensus_queue.cc:237] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } }
I20240503 02:13:12.980410 7990 raft_consensus.cc:2454] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f0228ec6c64da5b5d84fa3ebe0ade1 in term 1.
I20240503 02:13:12.981158 7900 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.981343 7757 heartbeater.cc:503] Master 127.4.17.124:34141 was elected leader, sending a full tablet report...
I20240503 02:13:12.982940 8216 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:12.990844 8216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:41621; Asking this server to re-register.
I20240503 02:13:12.991657 7757 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.993356 8216 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:13.065675 8112 consensus_queue.cc:1035] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.074259 8294 consensus_queue.cc:1035] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.093096 8294 consensus_queue.cc:1035] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.100346 8293 consensus_queue.cc:1035] T 27095a9813314d12804fa01ff5532b3c P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.164745 8114 consensus_queue.cc:1035] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.173928 8302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f2855cc86e414615a261b7a7d4bd3293)
I20240503 02:13:13.174151 8302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:13.175031 8302 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:13.180188 7442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240503 02:13:13.183104 7442 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:13.182951 8300 consensus_queue.cc:1035] T be53b36465e2461dbea23d101f957318 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.185928 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" is_pre_election: true
I20240503 02:13:13.186610 8226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 3.
I20240503 02:13:13.187173 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:13.187587 8302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240503 02:13:13.187729 8302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Starting leader election (detected failure of leader f2855cc86e414615a261b7a7d4bd3293)
I20240503 02:13:13.187841 8302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:13:13.190250 8302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:13.190927 8302 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:13.191808 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
I20240503 02:13:13.191998 8226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 3 FOLLOWER]: Advancing to term 5
W20240503 02:13:13.192534 7442 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:13.196796 8226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 5.
I20240503 02:13:13.197327 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:13.197676 8302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:13:13.198170 8302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 LEADER]: Becoming Leader. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Running, Role: LEADER
I20240503 02:13:13.198730 8302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:13.200610 8307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.200871 8307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:13.201526 8309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:13.204463 8309 catalog_manager.cc:669] Loaded metadata for table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]
I20240503 02:13:13.205123 8309 catalog_manager.cc:669] Loaded metadata for table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]
I20240503 02:13:13.205608 8309 catalog_manager.cc:669] Loaded metadata for table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]
I20240503 02:13:13.206220 8309 catalog_manager.cc:669] Loaded metadata for table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]
I20240503 02:13:13.206756 8309 catalog_manager.cc:669] Loaded metadata for table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]
I20240503 02:13:13.207262 8309 catalog_manager.cc:669] Loaded metadata for table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]
I20240503 02:13:13.207660 8309 catalog_manager.cc:669] Loaded metadata for table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]
I20240503 02:13:13.208056 8309 catalog_manager.cc:669] Loaded metadata for table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]
I20240503 02:13:13.208395 8309 catalog_manager.cc:669] Loaded metadata for table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]
I20240503 02:13:13.210283 8309 tablet_loader.cc:96] loaded metadata for tablet 0e9a18032b2f4b48a9b99910fa28f16f (table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e])
I20240503 02:13:13.210649 8309 tablet_loader.cc:96] loaded metadata for tablet 27095a9813314d12804fa01ff5532b3c (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c])
I20240503 02:13:13.210961 8309 tablet_loader.cc:96] loaded metadata for tablet 7099ff89ceb544119b317319a4d0aa23 (table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657])
I20240503 02:13:13.211249 8309 tablet_loader.cc:96] loaded metadata for tablet 8a8dca752b6947f8a891a9af6a58fdf4 (table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583])
I20240503 02:13:13.211547 8309 tablet_loader.cc:96] loaded metadata for tablet be53b36465e2461dbea23d101f957318 (table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f])
I20240503 02:13:13.211830 8309 tablet_loader.cc:96] loaded metadata for tablet d818c02e5f8c4bc8816a5cc702cf0b78 (table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52])
I20240503 02:13:13.212118 8309 tablet_loader.cc:96] loaded metadata for tablet e8d80f4253024847b86b0fdc6f31e175 (table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334])
I20240503 02:13:13.212396 8309 tablet_loader.cc:96] loaded metadata for tablet f3014429e2a54d01ba806f9b26e58485 (table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c])
I20240503 02:13:13.212688 8309 tablet_loader.cc:96] loaded metadata for tablet fb1acfaf1d3d4673b89f0de854e60d0e (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d])
I20240503 02:13:13.212982 8309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:13.214087 8309 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:13.214174 8309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:13.215481 8309 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:13.216393 8309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Loaded TSK: 0
I20240503 02:13:13.217368 8309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:13.218211 8309 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:13.219048 8309 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:13.256161 7452 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
I20240503 02:13:13.256515 7452 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
I20240503 02:13:13.255686 7456 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_f75d2fc6f4274e8bacbd0c7fe5ad767d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:13.255769 7453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_a096e9a32425495bb956cf7da2231847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:13.255707 7455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_1690fbbb7f924bc4991017df16cd36de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:13.256454 7450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_66cae47afa014a96a98625b9f519f2a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:13.257234 7448 master_service.cc:946] ReplaceTablet: received request to replace tablet 05406776807c49419ee4067df0b023a9 from {username='slave'} at 127.0.0.1:59606
I20240503 02:13:13.257145 7447 master_service.cc:946] ReplaceTablet: received request to replace tablet 27095a9813314d12804fa01ff5532b3c from {username='slave'} at 127.0.0.1:59606
I20240503 02:13:13.258364 7449 master_service.cc:946] ReplaceTablet: received request to replace tablet 27095a9813314d12804fa01ff5532b3c from {username='slave'} at 127.0.0.1:59606
I20240503 02:13:13.258729 7454 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_ddb14bf51d034e819a0d6b707439b04c" } modify_external_catalogs: true
I20240503 02:13:13.258997 7454 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_ddb14bf51d034e819a0d6b707439b04c" } modify_external_catalogs: true
I20240503 02:13:13.259265 7448 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a8dca752b6947f8a891a9af6a58fdf4 from {username='slave'} at 127.0.0.1:59606
I20240503 02:13:13.260389 7451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_7f03d6519166457e8a6519e2133ad38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20240503 02:13:13.267017 7623 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cba0c48d8df441209b09207d54435b46
*** Check failure stack trace: ***
I20240503 02:13:13.267757 8226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Refusing update from remote peer 96754aa0950c421b84eb59e6c6fbd2fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 02:13:13.268617 8307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:13.268949 8307 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Peer 0fefe1a587244be2b2dec28f7fa3f9d9 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
@ 0x7fc1f2abadcd google::LogMessage::Fail() at ??:0
@ 0x7fc1f2abeb93 google::LogMessage::SendToLog() at ??:0
W20240503 02:13:13.271463 7442 consensus_peers.cc:485] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe -> Peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Couldn't send request to peer f2855cc86e414615a261b7a7d4bd3293. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.272388 8226 pending_rounds.cc:76] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Aborting all ops after (but not including) 4
I20240503 02:13:13.272526 8226 pending_rounds.cc:97] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 02:13:13.272655 8226 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7fc1f2aba7cc google::LogMessage::Flush() at ??:0
@ 0x7fc1f2abaa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fc208cd057e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fc208ccf261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240503 02:13:13.293606 8315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.293967 8315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:13.294363 8317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.294699 8317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:13.295003 8312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.295320 8312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:13.295542 8312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.295724 8312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:13.298486 8310 mvcc.cc:205] Tried to move back new op lower bound from 7023421002807697408 to 7023421002546515968. Current Snapshot: MvccSnapshot[applied={T|T < 7023421002807697408}]
@ 0x7fc208cd62bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20240503 02:13:13.305825 8313 mvcc.cc:205] Tried to move back new op lower bound from 7023421002816868352 to 7023420988157222912. Current Snapshot: MvccSnapshot[applied={T|T < 7023421002816868352}]
@ 0x7fc208cd6139 std::_Function_handler<>::_M_invoke() at ??:0
I20240503 02:13:13.312659 7448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet 8a8dca752b6947f8a891a9af6a58fdf4
I20240503 02:13:13.312983 8307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:13.313256 8307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:13.313717 7447 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet 27095a9813314d12804fa01ff5532b3c
I20240503 02:13:13.314268 7449 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_0ec6c406990740d085be8679baa773d7" } modify_external_catalogs: true
I20240503 02:13:13.314551 7449 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_0ec6c406990740d085be8679baa773d7" } modify_external_catalogs: true
I20240503 02:13:13.314733 7448 catalog_manager.cc:6554] ReplaceTablet: tablet 8a8dca752b6947f8a891a9af6a58fdf4 deleted and replaced by tablet 11ef52dfabf6432aac84478cf39aaa7c
I20240503 02:13:13.314956 7448 master_service.cc:946] ReplaceTablet: received request to replace tablet 0e9a18032b2f4b48a9b99910fa28f16f from {username='slave'} at 127.0.0.1:59606
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 02:13:13.314738 7447 catalog_manager.cc:6554] ReplaceTablet: tablet 27095a9813314d12804fa01ff5532b3c deleted and replaced by tablet 63774b05816440d79a034425d863e045
I20240503 02:13:13.325027 7447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59606:
name: "default.Table_b5dab19d77e74121985b7124ff4f1716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:13.330935 8293 consensus_queue.cc:1035] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7fc1f57aa01e kudu::Thread::SuperviseThread() at ??:0
I20240503 02:13:13.333745 7448 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet 0e9a18032b2f4b48a9b99910fa28f16f
I20240503 02:13:13.336892 7448 catalog_manager.cc:6554] ReplaceTablet: tablet 0e9a18032b2f4b48a9b99910fa28f16f deleted and replaced by tablet ef243ef5a3024262bbc3c57a5bf3830b
I20240503 02:13:13.337347 7448 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
I20240503 02:13:13.337600 7448 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59606:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
@ 0x7fc1f8b2f6db start_thread at ??:0
@ 0x7fc1f196d71f clone at ??:0
I20240503 02:13:13.361128 8294 consensus_queue.cc:1035] T 0e9a18032b2f4b48a9b99910fa28f16f P c67f5cc3ea7a4529a01ede7d54fe4710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:13.550871 7646 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:13.551707 8199 connection.cc:535] server connection from 127.0.0.1:58430 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:13.551916 8329 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:13.317866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.4.17.66:33615)
0503 02:13:13.360067 (+ 42201us) server_negotiation.cc:206] Beginning negotiation
0503 02:13:13.360073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 02:13:13.360096 (+ 23us) server_negotiation.cc:407] Connection header received
0503 02:13:13.360158 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 02:13:13.360163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 02:13:13.360249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 02:13:13.360354 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 02:13:13.550871 (+190517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40414: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":42047,"thread_start_us":41914,"threads_started":1}
W20240503 02:13:13.551934 8330 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:13.317975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.4.17.67:41095)
0503 02:13:13.329050 (+ 11075us) server_negotiation.cc:206] Beginning negotiation
0503 02:13:13.329060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0503 02:13:13.329090 (+ 30us) server_negotiation.cc:407] Connection header received
0503 02:13:13.329181 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 02:13:13.329190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 02:13:13.329294 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 02:13:13.329438 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 02:13:13.340574 (+ 11136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 02:13:13.353782 (+ 13208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 02:13:13.550868 (+197086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58400: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":10890,"thread_start_us":21538,"threads_started":1}
W20240503 02:13:13.552551 8046 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:13.553257 8046 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.125:45323)
W20240503 02:13:13.553845 8046 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:13.554013 8046 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:13.553983 8328 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:13.317611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.4.17.65:45901)
0503 02:13:13.355746 (+ 38135us) server_negotiation.cc:206] Beginning negotiation
0503 02:13:13.355755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 02:13:13.355787 (+ 32us) server_negotiation.cc:407] Connection header received
0503 02:13:13.355884 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 02:13:13.355893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 02:13:13.356014 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 02:13:13.356147 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 02:13:13.552908 (+196761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33546: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":37866,"thread_start_us":37732,"threads_started":1}
W20240503 02:13:13.554271 7789 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:13.554680 8046 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.125:45323)
I20240503 02:13:13.559044 8300 consensus_queue.cc:1035] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.566280 8300 consensus_queue.cc:1035] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.754658 8344 consensus_queue.cc:1035] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.760046 8342 consensus_queue.cc:1035] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.852377 8349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:13.852718 8349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:13.854967 8349 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:13.856052 8198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:13.857549 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:13.857772 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:13.857923 8198 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:13.858317 8349 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:13:14.532255 8362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:14.532500 8362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:14.533303 8362 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:14.535617 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:14.535792 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:14.535998 8197 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:14.536321 8362 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:13:15.293991 8373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:15.294212 8373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:15.295006 8373 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:15.298012 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:15.298218 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:15.298409 8197 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:15.298872 8373 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 02:13:15.805770 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--webserver_interface=127.4.17.125
--webserver_port=46239
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:16.170341 8383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:16.170706 8383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:16.170794 8383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:16.179071 8383 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:16.179203 8383 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:16.179247 8383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:16.179306 8383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:16.179358 8383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:16.179425 8383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:16.189451 8383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.17.125
--webserver_port=46239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:16.190932 8383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:16.192824 8383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:16.205368 8396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:16.205929 8395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:16.207213 8398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:16.208575 8383 server_base.cc:1034] running on GCE node
I20240503 02:13:16.209558 8383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:16.211162 8383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:16.212410 8383 hybrid_clock.cc:648] HybridClock initialized: now 1714702396212324 us; error 59 us; skew 500 ppm
I20240503 02:13:16.212788 8383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:16.215875 8383 webserver.cc:460] Webserver started at http://127.4.17.125:46239/ using document root <none> and password file <none>
I20240503 02:13:16.216660 8383 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:16.216800 8383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:16.222685 8383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 02:13:16.225869 8404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:16.227526 8383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 02:13:16.227716 8383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:16.228638 8383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:16.245559 8383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:16.246527 8383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:16.246814 8383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:16.265864 8455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.125:45323 every 8 connection(s)
I20240503 02:13:16.265908 8383 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.125:45323
I20240503 02:13:16.267745 8383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
I20240503 02:13:16.274783 8456 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:16.274919 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 8383
I20240503 02:13:16.279493 8456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap starting.
I20240503 02:13:16.294595 8456 log.cc:824] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:16.323696 8456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=35 ignored=0} inserts{seen=26 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:13:16.324363 8456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap complete.
I20240503 02:13:16.331076 8456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:16.334118 8456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.334851 8456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 5.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:16.335683 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:16.335958 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:16.337038 8456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:16.347119 8383 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111),
I20240503 02:13:16.359686 8383 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:16.360898 8475 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:16.361058 8475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: loading cluster ID for follower catalog manager: success
I20240503 02:13:16.365772 8475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: acquiring CA information for follower catalog manager: success
I20240503 02:13:16.368530 8475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:16.420153 8038 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:16.424171 8421 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:58825; Asking this server to re-register.
I20240503 02:13:16.425132 8038 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:16.427009 8421 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:16.450017 7759 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:16.451925 8421 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:37075; Asking this server to re-register.
I20240503 02:13:16.452539 7759 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:16.453866 8421 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:16.539898 7902 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:16.542409 8421 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:59001; Asking this server to re-register.
I20240503 02:13:16.543182 7902 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:16.544961 8421 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:16.756564 8461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.756850 8461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:16.758849 8461 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:16.760049 8407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:16.770056 8407 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:16.775163 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" is_pre_election: true
I20240503 02:13:16.775386 8226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 5.
I20240503 02:13:16.776052 8406 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:16.776463 8461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240503 02:13:16.776685 8461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.776798 8461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:13:16.780506 8461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:16.781302 8461 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:16.781950 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9"
I20240503 02:13:16.782159 8226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:13:16.782996 8407 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:16.784365 8226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 6.
I20240503 02:13:16.784869 8406 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:16.785230 8461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:13:16.785715 8461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 LEADER]: Becoming Leader. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Running, Role: LEADER
I20240503 02:13:16.786228 8461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.36, Last appended by leader: 36, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:16.788990 8490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 6 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:16.789291 8490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:17.366835 8226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Refusing update from remote peer 96754aa0950c421b84eb59e6c6fbd2fe: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 6 index: 37. (index mismatch)
I20240503 02:13:17.367749 8494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 02:13:17.370551 8496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 6 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:17.370806 8496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:17.373806 8498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:17.374184 8498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:17.375037 8496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:17.375277 8496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:13:17.376730 8407 consensus_peers.cc:485] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe -> Peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Couldn't send request to peer f2855cc86e414615a261b7a7d4bd3293. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:17.435029 8492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:17.437911 8492 catalog_manager.cc:669] Loaded metadata for table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]
I20240503 02:13:17.438452 8492 catalog_manager.cc:669] Loaded metadata for table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]
I20240503 02:13:17.438835 8492 catalog_manager.cc:669] Loaded metadata for table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]
I20240503 02:13:17.439194 8492 catalog_manager.cc:669] Loaded metadata for table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]
I20240503 02:13:17.439523 8492 catalog_manager.cc:669] Loaded metadata for table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]
I20240503 02:13:17.440059 8492 catalog_manager.cc:669] Loaded metadata for table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]
I20240503 02:13:17.440465 8492 catalog_manager.cc:669] Loaded metadata for table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]
I20240503 02:13:17.440850 8492 catalog_manager.cc:669] Loaded metadata for table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]
I20240503 02:13:17.441208 8492 catalog_manager.cc:669] Loaded metadata for table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]
I20240503 02:13:17.443301 8492 tablet_loader.cc:96] loaded metadata for tablet 11ef52dfabf6432aac84478cf39aaa7c (table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583])
I20240503 02:13:17.443782 8492 tablet_loader.cc:96] loaded metadata for tablet 63774b05816440d79a034425d863e045 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c])
I20240503 02:13:17.444103 8492 tablet_loader.cc:96] loaded metadata for tablet 7099ff89ceb544119b317319a4d0aa23 (table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657])
I20240503 02:13:17.444734 8492 tablet_loader.cc:96] loaded metadata for tablet be53b36465e2461dbea23d101f957318 (table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f])
I20240503 02:13:17.445178 8492 tablet_loader.cc:96] loaded metadata for tablet d818c02e5f8c4bc8816a5cc702cf0b78 (table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52])
I20240503 02:13:17.445647 8492 tablet_loader.cc:96] loaded metadata for tablet e8d80f4253024847b86b0fdc6f31e175 (table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334])
I20240503 02:13:17.446072 8492 tablet_loader.cc:96] loaded metadata for tablet ef243ef5a3024262bbc3c57a5bf3830b (table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e])
I20240503 02:13:17.446530 8492 tablet_loader.cc:96] loaded metadata for tablet f3014429e2a54d01ba806f9b26e58485 (table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c])
I20240503 02:13:17.446892 8492 tablet_loader.cc:96] loaded metadata for tablet fb1acfaf1d3d4673b89f0de854e60d0e (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d])
I20240503 02:13:17.447173 8492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:17.448230 8492 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:17.448307 8492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:17.449875 8492 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:17.450788 8492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Loaded TSK: 0
I20240503 02:13:17.451690 8492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:17.452648 8492 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:17.453564 8492 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:17.514772 8421 master_service.cc:946] ReplaceTablet: received request to replace tablet f3014429e2a54d01ba806f9b26e58485 from {username='slave'} at 127.0.0.1:42674
I20240503 02:13:17.515894 8420 master_service.cc:946] ReplaceTablet: received request to replace tablet fb1acfaf1d3d4673b89f0de854e60d0e from {username='slave'} at 127.0.0.1:42674
I20240503 02:13:17.516480 8414 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_fa460008f83a4dc499b1ee88ee0e7925" } new_table_name: "default.Table_93b16ebe2e78419f85f3016868132e3e" modify_external_catalogs: true
I20240503 02:13:17.518033 8415 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_81067e0e8faa4902a7f68b688523e3e4" } new_table_name: "default.Table_46f2cd6cae304057bb0f555ab6d065fc" modify_external_catalogs: true
I20240503 02:13:17.518477 8413 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_830f8efb3147411b8c2863df686abe6c" } new_table_name: "default.Table_46c477c0b6584ff8a2279dc7fba52e9c" modify_external_catalogs: true
I20240503 02:13:17.519160 8419 master_service.cc:946] ReplaceTablet: received request to replace tablet 0e9a18032b2f4b48a9b99910fa28f16f from {username='slave'} at 127.0.0.1:42674
I20240503 02:13:17.520211 8416 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42674:
name: "default.Table_721855476892497ba8ae9b4ad7930e99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:17.521157 8418 master_service.cc:946] ReplaceTablet: received request to replace tablet d818c02e5f8c4bc8816a5cc702cf0b78 from {username='slave'} at 127.0.0.1:42674
I20240503 02:13:17.521497 8417 master_service.cc:946] ReplaceTablet: received request to replace tablet d818c02e5f8c4bc8816a5cc702cf0b78 from {username='slave'} at 127.0.0.1:42674
I20240503 02:13:17.522194 8419 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42674:
name: "default.Table_bcc90e6508b24abfafc68d6715194f7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:17.523792 8412 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42674:
name: "default.Table_47870538eb6041179084ffe5cf79566b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20240503 02:13:17.525164 8474 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cba0c48d8df441209b09207d54435b46
*** Check failure stack trace: ***
@ 0x7f6132f16dcd google::LogMessage::Fail() at ??:0
@ 0x7f6132f1ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6132f167cc google::LogMessage::Flush() at ??:0
@ 0x7f6132f16a11 google::LogMessage::~LogMessage() at ??:0
I20240503 02:13:17.541712 8421 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet f3014429e2a54d01ba806f9b26e58485
I20240503 02:13:17.547060 8420 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:17.547809 8420 catalog_manager.cc:6554] ReplaceTablet: tablet fb1acfaf1d3d4673b89f0de854e60d0e deleted and replaced by tablet a3907eb584054a73973b35efc23a5ce8
@ 0x7f614912c57e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240503 02:13:17.548512 8420 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42674:
name: "default.Table_a15ad42e445d4011a7ce7a358eb31e66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:17.555034 8421 catalog_manager.cc:6554] ReplaceTablet: tablet f3014429e2a54d01ba806f9b26e58485 deleted and replaced by tablet 366a898b86db46baa61ed152e98305bb
@ 0x7f614912b261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240503 02:13:17.562135 8421 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_bd10243fcd8e4436952edc7daf12e223" } new_table_name: "default.Table_ffe0555a64a74d05b9816a5be84d179c" modify_external_catalogs: true
@ 0x7f61491322bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f6149132139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6135c0601e kudu::Thread::SuperviseThread() at ??:0
I20240503 02:13:17.575552 8417 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Sending DeleteTablet for 3 replicas of tablet d818c02e5f8c4bc8816a5cc702cf0b78
I20240503 02:13:17.576289 8417 catalog_manager.cc:6554] ReplaceTablet: tablet d818c02e5f8c4bc8816a5cc702cf0b78 deleted and replaced by tablet 3e712201f5fc45408a06c86633dfb8b0
I20240503 02:13:17.576848 8417 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42674:
name: "default.Table_7f03d6519166457e8a6519e2133ad38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:17.577960 8418 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
I20240503 02:13:17.578270 8418 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42674:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
@ 0x7f6138f8b6db start_thread at ??:0
I20240503 02:13:17.579592 7969 tablet_service.cc:1514] Processing DeleteTablet for tablet f3014429e2a54d01ba806f9b26e58485 with delete_type TABLET_DATA_DELETED (replaced by tablet 366a898b86db46baa61ed152e98305bb) from {username='slave'} at 127.0.0.1:46000
I20240503 02:13:17.580133 7971 tablet_service.cc:1514] Processing DeleteTablet for tablet fb1acfaf1d3d4673b89f0de854e60d0e with delete_type TABLET_DATA_DELETED (replaced by tablet a3907eb584054a73973b35efc23a5ce8) from {username='slave'} at 127.0.0.1:46000
I20240503 02:13:17.580166 7970 tablet_service.cc:1467] Processing CreateTablet for tablet 366a898b86db46baa61ed152e98305bb (DEFAULT_TABLE table=default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.580193 7967 tablet_service.cc:1514] Processing DeleteTablet for tablet d818c02e5f8c4bc8816a5cc702cf0b78 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e712201f5fc45408a06c86633dfb8b0) from {username='slave'} at 127.0.0.1:46000
@ 0x7f6131dc971f clone at ??:0
I20240503 02:13:17.580776 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366a898b86db46baa61ed152e98305bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.582897 7968 tablet_service.cc:1467] Processing CreateTablet for tablet a3907eb584054a73973b35efc23a5ce8 (DEFAULT_TABLE table=default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.583469 8518 tablet_replica.cc:330] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: stopping tablet replica
I20240503 02:13:17.583539 7968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3907eb584054a73973b35efc23a5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.583765 8518 raft_consensus.cc:2229] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.584455 8518 raft_consensus.cc:2258] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.585218 8518 ts_tablet_manager.cc:1875] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.592500 8518 ts_tablet_manager.cc:1888] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:13:17.592672 8518 log.cc:1196] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/f3014429e2a54d01ba806f9b26e58485
I20240503 02:13:17.593243 8518 ts_tablet_manager.cc:1909] T f3014429e2a54d01ba806f9b26e58485 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting consensus metadata
I20240503 02:13:17.593554 8520 tablet_bootstrap.cc:492] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:17.594625 8518 tablet_replica.cc:330] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: stopping tablet replica
I20240503 02:13:17.594905 8518 raft_consensus.cc:2229] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:17.595520 8518 raft_consensus.cc:2258] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.595713 8520 tablet_bootstrap.cc:654] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.596204 8518 ts_tablet_manager.cc:1875] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.601455 8520 tablet_bootstrap.cc:492] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:17.601649 8520 ts_tablet_manager.cc:1367] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:13:17.602648 8520 raft_consensus.cc:348] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:17.602901 8520 raft_consensus.cc:374] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.603008 8520 raft_consensus.cc:729] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.603281 8520 consensus_queue.cc:260] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:17.604075 8518 ts_tablet_manager.cc:1888] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:13:17.604228 8518 log.cc:1196] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:17.605639 8520 ts_tablet_manager.cc:1398] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:17.606736 8520 tablet_bootstrap.cc:492] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:17.606803 8518 ts_tablet_manager.cc:1909] T fb1acfaf1d3d4673b89f0de854e60d0e P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting consensus metadata
I20240503 02:13:17.608665 8520 tablet_bootstrap.cc:654] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.609828 8518 tablet_replica.cc:330] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: stopping tablet replica
I20240503 02:13:17.610038 8518 raft_consensus.cc:2229] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.610267 8518 raft_consensus.cc:2258] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.610915 8518 ts_tablet_manager.cc:1875] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.617689 8520 tablet_bootstrap.cc:492] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:17.617895 8520 ts_tablet_manager.cc:1367] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 02:13:17.618453 8518 ts_tablet_manager.cc:1888] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:13:17.618595 8518 log.cc:1196] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting WAL directory at /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/ts-2/wal/wals/d818c02e5f8c4bc8816a5cc702cf0b78
I20240503 02:13:17.619096 8518 ts_tablet_manager.cc:1909] T d818c02e5f8c4bc8816a5cc702cf0b78 P c67f5cc3ea7a4529a01ede7d54fe4710: Deleting consensus metadata
I20240503 02:13:17.618932 8520 raft_consensus.cc:348] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:17.619217 8520 raft_consensus.cc:374] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.619300 8520 raft_consensus.cc:729] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.619724 8520 consensus_queue.cc:260] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:17.620405 8520 ts_tablet_manager.cc:1398] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 02:13:17.693341 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:17.745929 8147 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111),
I20240503 02:13:17.746106 8147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:13:17.780195 8517 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:17.546236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.4.17.65:45901)
0503 02:13:17.563509 (+ 17273us) server_negotiation.cc:206] Beginning negotiation
0503 02:13:17.563517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 02:13:17.779940 (+216423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47728: BlockingRecv error: recv got EOF from 127.0.0.1:47728 (error 108)
Metrics: {"server-negotiator.queue_time_us":17108,"thread_start_us":17285,"threads_started":1}
W20240503 02:13:17.780300 8515 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:17.545260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.4.17.66:33615)
0503 02:13:17.582502 (+ 37242us) server_negotiation.cc:206] Beginning negotiation
0503 02:13:17.582509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 02:13:17.582534 (+ 25us) server_negotiation.cc:407] Connection header received
0503 02:13:17.582628 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 02:13:17.582635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 02:13:17.582721 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 02:13:17.582843 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 02:13:17.779973 (+197130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51826: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":37085,"thread_start_us":37756,"threads_started":1}
W20240503 02:13:17.780866 8046 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:17.780906 7925 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:17.781180 7925 connection.cc:535] server connection from 127.0.0.1:46000 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:17.781638 8046 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.125:45323)
W20240503 02:13:17.781960 8046 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:17.782178 8046 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:17.782379 8046 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.125:45323)
I20240503 02:13:17.857659 8495 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.857867 8495 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:17.858600 8495 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:17.859697 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:17.859941 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:17.860077 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:17.860339 8495 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:17.959822 7759 heartbeater.cc:503] Master 127.4.17.125:45323 was elected leader, sending a full tablet report...
I20240503 02:13:18.057278 7902 heartbeater.cc:503] Master 127.4.17.125:45323 was elected leader, sending a full tablet report...
I20240503 02:13:18.076732 8535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:18.076977 8535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:18.077986 8535 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:18.079499 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:18.079670 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:18.079807 8198 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:18.080075 8535 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240503 02:13:18.190048 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.263447 8495 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.263677 8495 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:18.264418 8495 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:18.265506 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:18.265754 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:18.265909 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:18.266207 8495 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:18.759841 8542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:18.760017 8542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:18.760694 8542 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:18.762750 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:18.762993 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:13:18.763141 8197 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:18.763417 8542 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240503 02:13:19.418542 8539 raft_consensus.cc:482] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c67f5cc3ea7a4529a01ede7d54fe4710)
I20240503 02:13:19.418758 8539 raft_consensus.cc:504] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:19.419477 8539 leader_election.cc:288] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:19.420128 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" is_pre_election: true
W20240503 02:13:19.420610 7791 leader_election.cc:341] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:19.420873 7790 leader_election.cc:302] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710
I20240503 02:13:19.421190 8539 raft_consensus.cc:2726] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:19.595821 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--webserver_interface=127.4.17.125
--webserver_port=46239
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:13:19.698971 8557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:19.699208 8557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:19.699985 8557 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:19.700987 8198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:13:19.702379 8197 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:19.702527 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:19.702703 8198 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9; no voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:19.703058 8557 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240503 02:13:19.794821 8548 raft_consensus.cc:482] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c67f5cc3ea7a4529a01ede7d54fe4710)
I20240503 02:13:19.795228 8548 raft_consensus.cc:504] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:19.795974 8548 leader_election.cc:288] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:19.796664 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:19.797048 7855 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
W20240503 02:13:19.797129 7648 leader_election.cc:341] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:19.797580 7645 leader_election.cc:302] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters: c67f5cc3ea7a4529a01ede7d54fe4710
I20240503 02:13:19.797883 8548 raft_consensus.cc:2781] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:19.798003 8548 raft_consensus.cc:482] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c67f5cc3ea7a4529a01ede7d54fe4710)
I20240503 02:13:19.798101 8548 raft_consensus.cc:3037] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:19.800103 8548 raft_consensus.cc:504] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
I20240503 02:13:19.800727 8548 leader_election.cc:288] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 election: Requested vote from peers c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615), c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:19.801326 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1acfaf1d3d4673b89f0de854e60d0e" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:19.801568 7855 raft_consensus.cc:3037] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:19.801666 7648 leader_election.cc:341] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e
I20240503 02:13:19.804410 7855 raft_consensus.cc:2454] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 2.
I20240503 02:13:19.804879 7645 leader_election.cc:302] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters: c67f5cc3ea7a4529a01ede7d54fe4710
I20240503 02:13:19.805270 8548 raft_consensus.cc:2781] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:19.805457 8548 raft_consensus.cc:686] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [term 2 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:19.805759 8548 consensus_queue.cc:237] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } }
W20240503 02:13:19.989967 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:19.990286 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:19.990425 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:19.998847 8554 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:19.998978 8554 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:19.999027 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:19.999110 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:19.999178 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:19.999285 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:20.010386 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.17.125
--webserver_port=46239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:20.011902 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:20.013844 8554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:20.029383 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.030262 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.031239 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:20.032296 8554 server_base.cc:1034] running on GCE node
I20240503 02:13:20.033383 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:20.034969 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:20.036202 8554 hybrid_clock.cc:648] HybridClock initialized: now 1714702400036114 us; error 71 us; skew 500 ppm
I20240503 02:13:20.036676 8554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:20.039508 8554 webserver.cc:460] Webserver started at http://127.4.17.125:46239/ using document root <none> and password file <none>
I20240503 02:13:20.040411 8554 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:20.040581 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:20.047081 8554 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:20.050822 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.052631 8554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:20.052845 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:20.053932 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:20.077440 8554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:20.078389 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:20.078675 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:20.100811 8554 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.125:45323
I20240503 02:13:20.100847 8627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.125:45323 every 8 connection(s)
I20240503 02:13:20.102494 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
I20240503 02:13:20.108280 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 8554
I20240503 02:13:20.122790 8628 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:20.130896 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap starting.
I20240503 02:13:20.144412 7902 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:20.144681 7902 heartbeater.cc:503] Master 127.4.17.125:45323 was elected leader, sending a full tablet report...
I20240503 02:13:20.152853 8628 log.cc:824] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:20.186997 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=29 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.187614 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap complete.
I20240503 02:13:20.194361 8628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.195298 8628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.196072 8628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 6.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.197019 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:20.197299 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:20.198051 8628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:20.217365 8649 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:20.217543 8649 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: loading cluster ID for follower catalog manager: success
I20240503 02:13:20.230386 8649 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: acquiring CA information for follower catalog manager: success
I20240503 02:13:20.234186 8649 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:20.222877 8554 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111),
I20240503 02:13:20.234939 8554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:20.297461 8038 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:20.300034 8593 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:52823; Asking this server to re-register.
I20240503 02:13:20.300819 8038 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.302816 8593 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:20.314930 7759 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:20.315142 7759 heartbeater.cc:503] Master 127.4.17.125:45323 was elected leader, sending a full tablet report...
I20240503 02:13:20.317209 8593 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:38199; Asking this server to re-register.
I20240503 02:13:20.318063 7759 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.319689 8593 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:20.346712 8656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:20.346984 8656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.354486 8656 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
W20240503 02:13:20.366377 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:20.372709 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 42 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" is_pre_election: true
I20240503 02:13:20.373375 8603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 6.
I20240503 02:13:20.374117 8197 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:20.374461 8656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240503 02:13:20.374590 8656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:20.374691 8656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:13:20.377903 8656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.378793 8656 leader_election.cc:288] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323)
I20240503 02:13:20.379458 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 42 } } ignore_live_leader: false dest_uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
I20240503 02:13:20.379706 8603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 6 FOLLOWER]: Advancing to term 7
W20240503 02:13:20.382098 8198 leader_election.cc:334] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:20.385025 8603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fefe1a587244be2b2dec28f7fa3f9d9 in term 7.
I20240503 02:13:20.385552 8197 leader_election.cc:302] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe; no voters: f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:20.385890 8656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:13:20.386445 8656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [term 7 LEADER]: Becoming Leader. State: Replica: 0fefe1a587244be2b2dec28f7fa3f9d9, State: Running, Role: LEADER
I20240503 02:13:20.386920 8656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.388787 8661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 7 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:20.389180 8661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:20.389914 8663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:20.392685 8663 catalog_manager.cc:669] Loaded metadata for table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]
I20240503 02:13:20.393251 8663 catalog_manager.cc:669] Loaded metadata for table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]
I20240503 02:13:20.393646 8663 catalog_manager.cc:669] Loaded metadata for table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]
I20240503 02:13:20.393990 8663 catalog_manager.cc:669] Loaded metadata for table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]
I20240503 02:13:20.394325 8663 catalog_manager.cc:669] Loaded metadata for table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]
I20240503 02:13:20.394660 8663 catalog_manager.cc:669] Loaded metadata for table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]
I20240503 02:13:20.394995 8663 catalog_manager.cc:669] Loaded metadata for table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]
I20240503 02:13:20.395347 8663 catalog_manager.cc:669] Loaded metadata for table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]
I20240503 02:13:20.395673 8663 catalog_manager.cc:669] Loaded metadata for table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]
I20240503 02:13:20.397845 8663 tablet_loader.cc:96] loaded metadata for tablet 11ef52dfabf6432aac84478cf39aaa7c (table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583])
I20240503 02:13:20.398228 8663 tablet_loader.cc:96] loaded metadata for tablet 366a898b86db46baa61ed152e98305bb (table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c])
I20240503 02:13:20.398537 8663 tablet_loader.cc:96] loaded metadata for tablet 3e712201f5fc45408a06c86633dfb8b0 (table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52])
I20240503 02:13:20.398892 8663 tablet_loader.cc:96] loaded metadata for tablet 63774b05816440d79a034425d863e045 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c])
I20240503 02:13:20.399242 8663 tablet_loader.cc:96] loaded metadata for tablet 7099ff89ceb544119b317319a4d0aa23 (table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657])
I20240503 02:13:20.399590 8663 tablet_loader.cc:96] loaded metadata for tablet a3907eb584054a73973b35efc23a5ce8 (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d])
I20240503 02:13:20.399938 8663 tablet_loader.cc:96] loaded metadata for tablet be53b36465e2461dbea23d101f957318 (table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f])
I20240503 02:13:20.400440 8663 tablet_loader.cc:96] loaded metadata for tablet e8d80f4253024847b86b0fdc6f31e175 (table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334])
I20240503 02:13:20.400738 8663 tablet_loader.cc:96] loaded metadata for tablet ef243ef5a3024262bbc3c57a5bf3830b (table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e])
I20240503 02:13:20.401468 8663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:20.402556 8663 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:20.402633 8663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:20.403956 8663 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:20.404908 8663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9: Loaded TSK: 0
I20240503 02:13:20.405851 8663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:20.406819 8663 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:20.407829 8663 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:20.418695 8603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Refusing update from remote peer 0fefe1a587244be2b2dec28f7fa3f9d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 42. Preceding OpId from leader: term: 7 index: 44. (index mismatch)
W20240503 02:13:20.418859 8198 consensus_peers.cc:485] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 -> Peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Couldn't send request to peer f2855cc86e414615a261b7a7d4bd3293. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.419665 8661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20240503 02:13:20.425944 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fefe1a587244be2b2dec28f7fa3f9d9. Latest consensus state: current_term: 7 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:20.426327 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:20.428359 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:20.428716 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fefe1a587244be2b2dec28f7fa3f9d9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:20.430271 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:20.430539 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:20.445271 7970 tablet_service.cc:1467] Processing CreateTablet for tablet 3e712201f5fc45408a06c86633dfb8b0 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:20.445782 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e712201f5fc45408a06c86633dfb8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.446874 8215 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_82fc16f07bdb4d24bb75d0b04ffe9c29" } new_table_name: "default.Table_ec90921d196d413d8dc49d7ed15da51e" modify_external_catalogs: true
I20240503 02:13:20.448647 8214 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
I20240503 02:13:20.449024 8214 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_f1ff227bd5304d93a4c1ef884df9c987" } modify_external_catalogs: true
I20240503 02:13:20.454880 8212 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_0ec6c406990740d085be8679baa773d7" } modify_external_catalogs: true
I20240503 02:13:20.455202 8212 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_0ec6c406990740d085be8679baa773d7" } modify_external_catalogs: true
I20240503 02:13:20.455505 8211 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "default.Table_66cae47afa014a96a98625b9f519f2a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:20.455782 8209 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "default.Table_1690fbbb7f924bc4991017df16cd36de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:20.456310 8213 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "default.Table_b5dab19d77e74121985b7124ff4f1716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:20.456511 8210 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "default.Table_f75d2fc6f4274e8bacbd0c7fe5ad767d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:20.457201 8208 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_ddb14bf51d034e819a0d6b707439b04c" } modify_external_catalogs: true
I20240503 02:13:20.457487 8208 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_ddb14bf51d034e819a0d6b707439b04c" } modify_external_catalogs: true
I20240503 02:13:20.457368 8207 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "default.Table_721855476892497ba8ae9b4ad7930e99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:20.458811 8216 master_service.cc:946] ReplaceTablet: received request to replace tablet f3014429e2a54d01ba806f9b26e58485 from {username='slave'} at 127.0.0.1:58422
F20240503 02:13:20.464316 8284 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cba0c48d8df441209b09207d54435b46
*** Check failure stack trace: ***
@ 0x7fa97cb0adcd google::LogMessage::Fail() at ??:0
@ 0x7fa97cb0eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa97cb0a7cc google::LogMessage::Flush() at ??:0
@ 0x7fa97cb0aa11 google::LogMessage::~LogMessage() at ??:0
I20240503 02:13:20.473692 8673 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710: Bootstrap starting.
I20240503 02:13:20.476235 8673 tablet_bootstrap.cc:654] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.476461 8216 master_service.cc:946] ReplaceTablet: received request to replace tablet d818c02e5f8c4bc8816a5cc702cf0b78 from {username='slave'} at 127.0.0.1:58422
I20240503 02:13:20.477741 8216 master_service.cc:946] ReplaceTablet: received request to replace tablet d818c02e5f8c4bc8816a5cc702cf0b78 from {username='slave'} at 127.0.0.1:58422
I20240503 02:13:20.478443 8216 master_service.cc:946] ReplaceTablet: received request to replace tablet f3014429e2a54d01ba806f9b26e58485 from {username='slave'} at 127.0.0.1:58422
I20240503 02:13:20.479168 8216 master_service.cc:946] ReplaceTablet: received request to replace tablet fb1acfaf1d3d4673b89f0de854e60d0e from {username='slave'} at 127.0.0.1:58422
I20240503 02:13:20.479974 8216 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "default.Table_81067e0e8faa4902a7f68b688523e3e4" } new_table_name: "default.Table_46f2cd6cae304057bb0f555ab6d065fc" modify_external_catalogs: true
I20240503 02:13:20.485803 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 3e712201f5fc45408a06c86633dfb8b0 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:20.486342 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e712201f5fc45408a06c86633dfb8b0. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa992d2057e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240503 02:13:20.491588 7691 tablet_service.cc:1467] Processing CreateTablet for tablet 3e712201f5fc45408a06c86633dfb8b0 (DEFAULT_TABLE table=default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:20.492233 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e712201f5fc45408a06c86633dfb8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.494199 8673 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710: No bootstrap required, opened a new log
I20240503 02:13:20.494367 8673 ts_tablet_manager.cc:1367] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 02:13:20.495247 8673 raft_consensus.cc:348] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.495460 8673 raft_consensus.cc:374] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.495539 8673 raft_consensus.cc:729] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67f5cc3ea7a4529a01ede7d54fe4710, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.495867 8673 consensus_queue.cc:260] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.501124 8673 ts_tablet_manager.cc:1398] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
@ 0x7fa992d1f261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240503 02:13:20.508723 8677 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Bootstrap starting.
I20240503 02:13:20.511019 8677 tablet_bootstrap.cc:654] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.512872 8678 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Bootstrap starting.
I20240503 02:13:20.514425 8678 tablet_bootstrap.cc:654] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Neither blocks nor log segments found. Creating new log.
@ 0x7fa992d262bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20240503 02:13:20.522747 8677 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: No bootstrap required, opened a new log
I20240503 02:13:20.523012 8677 ts_tablet_manager.cc:1367] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240503 02:13:20.526304 8678 tablet_bootstrap.cc:492] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b: No bootstrap required, opened a new log
I20240503 02:13:20.526515 8678 ts_tablet_manager.cc:1367] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240503 02:13:20.527808 8678 raft_consensus.cc:348] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.528113 8678 raft_consensus.cc:374] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.528213 8678 raft_consensus.cc:729] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.528595 8678 consensus_queue.cc:260] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.529213 8678 ts_tablet_manager.cc:1398] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:13:20.524231 8677 raft_consensus.cc:348] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.531312 8677 raft_consensus.cc:374] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.531438 8677 raft_consensus.cc:729] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f0228ec6c64da5b5d84fa3ebe0ade1, State: Initialized, Role: FOLLOWER
@ 0x7fa992d26139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 02:13:20.536494 8677 consensus_queue.cc:260] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.543212 7902 heartbeater.cc:503] Master 127.4.17.125:45323 was elected leader, sending a full tablet report...
I20240503 02:13:20.545092 8677 ts_tablet_manager.cc:1398] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1: Time spent starting tablet: real 0.022s user 0.000s sys 0.007s
@ 0x7fa97f7fa01e kudu::Thread::SuperviseThread() at ??:0
I20240503 02:13:20.546247 8593 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:43671; Asking this server to re-register.
I20240503 02:13:20.547214 7902 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.549599 8593 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
@ 0x7fa982b7f6db start_thread at ??:0
@ 0x7fa97b9bd71f clone at ??:0
W20240503 02:13:20.741118 8046 connection.cc:535] client connection to 127.4.17.124:34141 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:20.741873 8046 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.124:34141)
W20240503 02:13:20.742574 8046 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.124:34141)
W20240503 02:13:20.742785 8046 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.124:34141)
W20240503 02:13:20.743101 8046 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.124:34141)
I20240503 02:13:20.840721 8659 raft_consensus.cc:482] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.840989 8659 raft_consensus.cc:504] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.841822 8659 leader_election.cc:288] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:20.842543 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e712201f5fc45408a06c86633dfb8b0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" is_pre_election: true
I20240503 02:13:20.842516 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e712201f5fc45408a06c86633dfb8b0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" is_pre_election: true
I20240503 02:13:20.842797 7990 raft_consensus.cc:2454] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:20.842849 7855 raft_consensus.cc:2454] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 0.
I20240503 02:13:20.843478 7648 leader_election.cc:302] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c67f5cc3ea7a4529a01ede7d54fe4710, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:20.843993 8659 raft_consensus.cc:2781] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.844148 8659 raft_consensus.cc:482] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.844259 8659 raft_consensus.cc:3037] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.846390 8659 raft_consensus.cc:504] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.847038 8659 leader_election.cc:288] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Requested vote from peers c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:20.847695 7990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e712201f5fc45408a06c86633dfb8b0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710"
I20240503 02:13:20.847740 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e712201f5fc45408a06c86633dfb8b0" candidate_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1"
I20240503 02:13:20.847945 7990 raft_consensus.cc:3037] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.847962 7855 raft_consensus.cc:3037] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.851006 7990 raft_consensus.cc:2454] T 3e712201f5fc45408a06c86633dfb8b0 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:20.850993 7855 raft_consensus.cc:2454] T 3e712201f5fc45408a06c86633dfb8b0 P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6a6a27cc814e0a86b1767df5f5a31b in term 1.
I20240503 02:13:20.851680 7645 leader_election.cc:302] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1; no voters:
I20240503 02:13:20.852078 8659 raft_consensus.cc:2781] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.852252 8659 raft_consensus.cc:686] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a6a27cc814e0a86b1767df5f5a31b, State: Running, Role: LEADER
I20240503 02:13:20.852644 8659 consensus_queue.cc:237] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:20.931245 7855 raft_consensus.cc:1261] T fb1acfaf1d3d4673b89f0de854e60d0e P c9f0228ec6c64da5b5d84fa3ebe0ade1 [term 2 FOLLOWER]: Refusing update from remote peer 6f6a6a27cc814e0a86b1767df5f5a31b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:13:20.931958 8659 consensus_queue.cc:1035] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:20.938937 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.977607 8702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:20.977878 8702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:20.979974 8702 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:20.980826 8579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:20.982095 8578 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:20.982283 8579 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:20.982448 8579 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:20.982813 8702 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:21.251634 8706 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.251827 8706 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:21.252655 8706 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:21.253774 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:21.253958 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:21.254063 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:21.254336 8706 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:21.606532 8714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:21.606735 8714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:21.607513 8714 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:21.609515 8579 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:21.609753 8578 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:21.609879 8578 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:21.610153 8714 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:21.711372 8706 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.711604 8706 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:21.712359 8706 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:21.713311 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:21.713474 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:21.713619 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:21.713919 8706 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:21.928176 8712 consensus_queue.cc:1035] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:21.935420 8711 consensus_queue.cc:1035] T 3e712201f5fc45408a06c86633dfb8b0 P 6f6a6a27cc814e0a86b1767df5f5a31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.396015 8730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fefe1a587244be2b2dec28f7fa3f9d9)
I20240503 02:13:22.396234 8730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:22.396950 8730 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:22.399164 8578 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:22.399453 8579 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:22.399598 8579 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:22.399885 8730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:22.596587 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 8554
I20240503 02:13:22.603964 8623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:22.714484 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 8554
I20240503 02:13:23.621083 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--webserver_interface=127.4.17.125
--webserver_port=46239
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:23.973582 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:23.973892 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:23.973984 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:23.982002 8755 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:23.982127 8755 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:23.982185 8755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:23.982244 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:23.982291 8755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:23.982352 8755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:23.992426 8755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.125:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.17.125
--webserver_port=46239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:23.994271 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:23.996246 8755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:24.008333 8765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:24.009608 8766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:24.010730 8768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:24.011904 8755 server_base.cc:1034] running on GCE node
I20240503 02:13:24.012976 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:24.014791 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:24.016009 8755 hybrid_clock.cc:648] HybridClock initialized: now 1714702404015940 us; error 40 us; skew 500 ppm
I20240503 02:13:24.016376 8755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:24.025301 8755 webserver.cc:460] Webserver started at http://127.4.17.125:46239/ using document root <none> and password file <none>
I20240503 02:13:24.026124 8755 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:24.026265 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:24.032248 8755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:13:24.035425 8774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:24.037009 8755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:13:24.037197 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
uuid: "96754aa0950c421b84eb59e6c6fbd2fe"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:24.038154 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:24.058861 8755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:24.059785 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:24.060098 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:24.079704 8755 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.125:45323
I20240503 02:13:24.079751 8825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.125:45323 every 8 connection(s)
I20240503 02:13:24.081465 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-1/data/info.pb
I20240503 02:13:24.088032 8826 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:24.088572 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 8755
I20240503 02:13:24.092425 8826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap starting.
I20240503 02:13:24.106794 8826 log.cc:824] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:24.129436 7759 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:24.148382 8826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=29 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.149160 8826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Bootstrap complete.
I20240503 02:13:24.159279 8826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:24.160549 8826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.161435 8826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 7.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:24.162328 8832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:24.162628 8832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:24.163662 8826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:24.180521 8846 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:24.180830 8846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: loading cluster ID for follower catalog manager: success
I20240503 02:13:24.183118 8755 master_runner.cc:186] Error getting master registration for 127.4.17.126:42485: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111),
I20240503 02:13:24.191251 8755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:24.192834 8846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: acquiring CA information for follower catalog manager: success
I20240503 02:13:24.196262 8846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:24.359069 8791 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:40447; Asking this server to re-register.
I20240503 02:13:24.360280 7759 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:24.362646 8791 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:24.455750 8038 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:24.457420 8791 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:42837; Asking this server to re-register.
I20240503 02:13:24.458110 8038 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:24.459764 8791 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:24.503049 7902 heartbeater.cc:347] Connected to a master server at 127.4.17.125:45323
I20240503 02:13:24.505174 8791 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:34687; Asking this server to re-register.
I20240503 02:13:24.506011 7902 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:24.507911 8791 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:24.639153 8832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:24.639470 8832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:24.642164 8832 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:24.642863 8777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:24.644402 8777 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:24.644620 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:24.644888 8776 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:24.645335 8832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:25.301820 8866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.302038 8866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:25.302809 8866 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:25.305073 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:25.305282 8777 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:25.305414 8777 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:25.305670 8866 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:26.375593 8884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.375835 8884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:26.376593 8884 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:26.378644 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:13:26.378857 8777 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
I20240503 02:13:26.378969 8777 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:26.379215 8884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240503 02:13:26.756071 8891 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.756258 8891 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:26.757287 8891 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:26.758221 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:26.758494 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:26.758620 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:26.758900 8891 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:26.821803 8891 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.821995 8891 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:26.822700 8891 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:26.823652 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:26.823860 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:26.823987 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:26.824298 8891 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:27.160149 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--webserver_interface=127.4.17.126
--webserver_port=39555
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 02:13:27.305385 8904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:27.305605 8904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.306370 8904 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:27.308784 8777 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485): Network error: Client connection negotiation failed: client connection to 127.4.17.126:42485: connect: Connection refused (error 111)
W20240503 02:13:27.308972 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:27.309084 8776 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, f2855cc86e414615a261b7a7d4bd3293
I20240503 02:13:27.309330 8904 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240503 02:13:27.408910 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:27.522306 8903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:27.522734 8903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:27.522858 8903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:27.535041 8903 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:27.535153 8903 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:27.535197 8903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:27.535254 8903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:27.535302 8903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:27.535365 8903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:27.544903 8903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=39555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:27.546272 8903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:27.547995 8903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:27.559845 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:27.560336 8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:27.561543 8919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:27.562618 8903 server_base.cc:1034] running on GCE node
I20240503 02:13:27.563599 8903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:27.565339 8903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:27.566589 8903 hybrid_clock.cc:648] HybridClock initialized: now 1714702407566521 us; error 35 us; skew 500 ppm
I20240503 02:13:27.566928 8903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:27.569428 8903 webserver.cc:460] Webserver started at http://127.4.17.126:39555/ using document root <none> and password file <none>
I20240503 02:13:27.570148 8903 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:27.570271 8903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:27.575956 8903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 02:13:27.579335 8925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:27.580840 8903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:27.581049 8903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:27.581955 8903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:27.598616 8903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:27.599539 8903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:27.599948 8903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:27.618971 8903 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:42485
I20240503 02:13:27.619035 8977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:42485 every 8 connection(s)
I20240503 02:13:27.620636 8903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:13:27.627351 8978 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:27.628608 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 8903
I20240503 02:13:27.631251 8978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap starting.
I20240503 02:13:27.644526 8978 log.cc:824] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:27.670102 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:27.674844 8978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=23 ignored=0} mutations{seen=17 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.675352 8978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap complete.
I20240503 02:13:27.682302 8978 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.683130 8978 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.683787 8978 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.684597 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:27.684902 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:27.685820 8978 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:27.693656 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:27.718379 9000 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:27.718595 9000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: loading cluster ID for follower catalog manager: success
I20240503 02:13:27.728873 9000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: acquiring CA information for follower catalog manager: success
I20240503 02:13:27.732822 9000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:27.743520 8903 master_runner.cc:186] Error getting master registration for 127.4.17.124:34141: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111),
I20240503 02:13:27.748605 8903 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:27.912001 9006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:27.912237 9006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.919965 7901 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:27.924026 9006 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:27.925072 8943 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:54753; Asking this server to re-register.
W20240503 02:13:27.925863 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:27.926791 7901 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:27.928972 8943 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:27.931304 8943 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:46129; Asking this server to re-register.
I20240503 02:13:27.933248 7758 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:27.935492 8943 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
W20240503 02:13:27.937033 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:27.936733 8953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 8 candidate_status { last_received { term: 7 index: 45 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293" is_pre_election: true
I20240503 02:13:27.937474 8953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 4.
I20240503 02:13:27.938222 8777 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9
I20240503 02:13:27.938587 9006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240503 02:13:27.938728 9006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:27.938827 9006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:13:27.942727 9006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.943423 9006 leader_election.cc:288] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 election: Requested vote from peers f2855cc86e414615a261b7a7d4bd3293 (127.4.17.126:42485), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:27.943979 8953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" candidate_term: 8 candidate_status { last_received { term: 7 index: 45 } } ignore_live_leader: false dest_uuid: "f2855cc86e414615a261b7a7d4bd3293"
I20240503 02:13:27.944193 8953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 4 FOLLOWER]: Advancing to term 8
W20240503 02:13:27.945070 8776 leader_election.cc:334] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:27.949954 8953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96754aa0950c421b84eb59e6c6fbd2fe in term 8.
I20240503 02:13:27.950526 8777 leader_election.cc:302] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96754aa0950c421b84eb59e6c6fbd2fe, f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9
I20240503 02:13:27.950902 9006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:13:27.951428 9006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [term 8 LEADER]: Becoming Leader. State: Replica: 96754aa0950c421b84eb59e6c6fbd2fe, State: Running, Role: LEADER
I20240503 02:13:27.951946 9006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 7.45, Last appended by leader: 45, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:27.954885 9011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 8 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:27.955246 9011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:27.955963 9013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:27.958988 9013 catalog_manager.cc:669] Loaded metadata for table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657]
I20240503 02:13:27.959735 9013 catalog_manager.cc:669] Loaded metadata for table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e]
I20240503 02:13:27.960536 9013 catalog_manager.cc:669] Loaded metadata for table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d]
I20240503 02:13:27.961241 9013 catalog_manager.cc:669] Loaded metadata for table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f]
I20240503 02:13:27.961905 9013 catalog_manager.cc:669] Loaded metadata for table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334]
I20240503 02:13:27.962543 9013 catalog_manager.cc:669] Loaded metadata for table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52]
I20240503 02:13:27.963194 9013 catalog_manager.cc:669] Loaded metadata for table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c]
I20240503 02:13:27.963873 9013 catalog_manager.cc:669] Loaded metadata for table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c]
I20240503 02:13:27.964545 9013 catalog_manager.cc:669] Loaded metadata for table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583]
I20240503 02:13:27.968189 9013 tablet_loader.cc:96] loaded metadata for tablet 11ef52dfabf6432aac84478cf39aaa7c (table default.table_fa460008f83a4dc499b1ee88ee0e7925 [id=f86e3f6464b74f0d84e2c1476e9fb583])
I20240503 02:13:27.968791 9013 tablet_loader.cc:96] loaded metadata for tablet 366a898b86db46baa61ed152e98305bb (table default.table_ddb14bf51d034e819a0d6b707439b04c [id=bb7ee1fde2f145feaf1ded54deff266c])
I20240503 02:13:27.969322 9013 tablet_loader.cc:96] loaded metadata for tablet 3e712201f5fc45408a06c86633dfb8b0 (table default.table_82fc16f07bdb4d24bb75d0b04ffe9c29 [id=ba32e3b5c4bf4aa8874f5afa875ccf52])
I20240503 02:13:27.969966 9013 tablet_loader.cc:96] loaded metadata for tablet 63774b05816440d79a034425d863e045 (table default.table_830f8efb3147411b8c2863df686abe6c [id=c74da947f58744958c70d8bb9a9d756c])
I20240503 02:13:27.970539 9013 tablet_loader.cc:96] loaded metadata for tablet 7099ff89ceb544119b317319a4d0aa23 (table default.table_bd10243fcd8e4436952edc7daf12e223 [id=26947ead47f4427ba253ee462c99b657])
I20240503 02:13:27.971092 9013 tablet_loader.cc:96] loaded metadata for tablet a3907eb584054a73973b35efc23a5ce8 (table default.table_81067e0e8faa4902a7f68b688523e3e4 [id=4ef7334f794b4e53b19ff587daf1455d])
I20240503 02:13:27.971668 9013 tablet_loader.cc:96] loaded metadata for tablet be53b36465e2461dbea23d101f957318 (table default.table_6c742fade1b9400f93b9e39783dbecc4 [id=5069a59fcea74f718a1b945cc20aab7f])
I20240503 02:13:27.972193 9013 tablet_loader.cc:96] loaded metadata for tablet e8d80f4253024847b86b0fdc6f31e175 (table default.table_0ec6c406990740d085be8679baa773d7 [id=9a298c9ff14e4f6babe912e4148e4334])
I20240503 02:13:27.972728 9013 tablet_loader.cc:96] loaded metadata for tablet ef243ef5a3024262bbc3c57a5bf3830b (table default.table_f1ff227bd5304d93a4c1ef884df9c987 [id=45b26215e3024493a872c4c7e0261d7e])
I20240503 02:13:27.973248 9013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:27.975901 9013 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:27.975999 9013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:27.978358 9013 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:27.979920 9013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe: Loaded TSK: 0
I20240503 02:13:27.981132 9013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:27.981895 8037 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:27.982580 9013 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240503 02:13:27.983701 9013 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240503 02:13:27.984697 8943 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:43463; Asking this server to re-register.
I20240503 02:13:27.985514 8037 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:27.987262 8943 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:28.098138 8789 master_service.cc:946] ReplaceTablet: received request to replace tablet 7099ff89ceb544119b317319a4d0aa23 from {username='slave'} at 127.0.0.1:40118
I20240503 02:13:28.098624 8790 master_service.cc:946] ReplaceTablet: received request to replace tablet be53b36465e2461dbea23d101f957318 from {username='slave'} at 127.0.0.1:40118
I20240503 02:13:28.099169 8791 master_service.cc:946] ReplaceTablet: received request to replace tablet 27095a9813314d12804fa01ff5532b3c from {username='slave'} at 127.0.0.1:40118
I20240503 02:13:28.100414 8788 master_service.cc:946] ReplaceTablet: received request to replace tablet 27095a9813314d12804fa01ff5532b3c from {username='slave'} at 127.0.0.1:40118
I20240503 02:13:28.100996 8786 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "default.Table_a096e9a32425495bb956cf7da2231847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:28.100391 8791 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "default.Table_47870538eb6041179084ffe5cf79566b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:28.101135 8788 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "default.Table_bcc90e6508b24abfafc68d6715194f7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:28.100540 8787 master_service.cc:946] ReplaceTablet: received request to replace tablet be53b36465e2461dbea23d101f957318 from {username='slave'} at 127.0.0.1:40118
I20240503 02:13:28.104017 8785 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40118:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
I20240503 02:13:28.104485 8785 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40118:
table { table_name: "default.Table_6c742fade1b9400f93b9e39783dbecc4" } modify_external_catalogs: true
I20240503 02:13:28.106165 8783 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40118:
table { table_name: "default.Table_bd10243fcd8e4436952edc7daf12e223" } new_table_name: "default.Table_ffe0555a64a74d05b9816a5be84d179c" modify_external_catalogs: true
I20240503 02:13:28.105572 8784 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "default.Table_7f03d6519166457e8a6519e2133ad38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:28.106704 8782 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "default.Table_a15ad42e445d4011a7ce7a358eb31e66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:13:28.109750 8953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Refusing update from remote peer 96754aa0950c421b84eb59e6c6fbd2fe: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 8 index: 47. (index mismatch)
W20240503 02:13:28.109934 8776 consensus_peers.cc:485] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe -> Peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Couldn't send request to peer 0fefe1a587244be2b2dec28f7fa3f9d9. Status: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.110704 9011 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96754aa0950c421b84eb59e6c6fbd2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 30, Time since last communication: 0.000s
F20240503 02:13:28.113029 8845 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cba0c48d8df441209b09207d54435b46
*** Check failure stack trace: ***
@ 0x7f0bd4266dcd google::LogMessage::Fail() at ??:0
@ 0x7f0bd426ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0bd42667cc google::LogMessage::Flush() at ??:0
@ 0x7f0bd4266a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0bea47c57e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f0bea47b261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f0bea4822bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f0bea482139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0bd6f5601e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0bda2db6db start_thread at ??:0
@ 0x7f0bd311971f clone at ??:0
I20240503 02:13:28.173425 9018 mvcc.cc:205] Tried to move back new op lower bound from 7023421020445106176 to 7023421002546515968. Current Snapshot: MvccSnapshot[applied={T|T < 7023421020445106176 or (T in {7023421020445106176})}]
I20240503 02:13:28.176077 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:28.176424 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:28.176867 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:28.177277 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:28.177238 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96754aa0950c421b84eb59e6c6fbd2fe. Latest consensus state: current_term: 8 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:28.177462 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:28.178867 9028 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:28.179198 9028 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:13:28.342144 8929 connection.cc:535] server connection from 127.0.0.1:45748 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:28.342446 8046 connection.cc:535] client connection to 127.4.17.125:45323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:28.343616 8046 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.17.125:45323)
W20240503 02:13:28.344008 8046 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:28.344308 8046 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.17.125:45323)
W20240503 02:13:28.344514 8046 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.17.125:45323)
I20240503 02:13:28.621277 9028 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:28.621493 9028 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:28.623428 9028 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:28.624645 8927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:28.627079 8927 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:28.627386 8927 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:28.627506 8927 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:28.627907 9028 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:29.372745 9051 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 96754aa0950c421b84eb59e6c6fbd2fe)
I20240503 02:13:29.373020 9051 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:29.373821 9051 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:29.376168 8927 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:29.376345 8927 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:29.376449 8927 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:29.376701 9051 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:30.239846 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 8903
I20240503 02:13:30.253119 8973 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:30.372710 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 8903
I20240503 02:13:30.975912 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--webserver_interface=127.4.17.126
--webserver_port=39555
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:31.362077 9078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:31.362428 9078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:31.362525 9078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:31.370743 9078 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:31.370903 9078 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:31.370968 9078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:31.371068 9078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:31.371155 9078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:31.371279 9078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:31.381587 9078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=39555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:31.383029 9078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:31.384994 9078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:31.397737 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:31.398787 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:31.400058 9089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:31.401499 9078 server_base.cc:1034] running on GCE node
I20240503 02:13:31.402494 9078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:31.404284 9078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:31.405580 9078 hybrid_clock.cc:648] HybridClock initialized: now 1714702411405519 us; error 56 us; skew 500 ppm
I20240503 02:13:31.406028 9078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:31.409530 9078 webserver.cc:460] Webserver started at http://127.4.17.126:39555/ using document root <none> and password file <none>
I20240503 02:13:31.410461 9078 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:31.410636 9078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:31.417948 9078 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:31.421850 9095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:31.423715 9078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:31.423956 9078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:31.425026 9078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:31.447321 9078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:31.448298 9078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:31.448612 9078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:31.469022 9078 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:42485
I20240503 02:13:31.469063 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:42485 every 8 connection(s)
I20240503 02:13:31.470867 9078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:13:31.478072 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 9078
I20240503 02:13:31.478340 9148 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:31.482936 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap starting.
I20240503 02:13:31.499008 9148 log.cc:824] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:31.525552 8037 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:31.526502 7901 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:31.550037 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=45 ignored=0} inserts{seen=29 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 02:13:31.550665 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap complete.
I20240503 02:13:31.557718 9148 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:31.561457 9148 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.562224 9148 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 8.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:31.563151 9157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:31.563443 9100 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:31.563545 9157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:31.563632 9100 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:31.564523 9148 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:31.564487 9100 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:31.565539 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:31.565697 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:31.565845 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:31.566165 9100 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:31.577836 9171 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:31.578025 9171 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: loading cluster ID for follower catalog manager: success
I20240503 02:13:31.583098 9171 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: acquiring CA information for follower catalog manager: success
I20240503 02:13:31.586900 9171 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:31.592832 9078 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:31.599583 9078 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:13:31.769462 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:31.777213 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:31.780429 9112 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:39477; Asking this server to re-register.
I20240503 02:13:31.781369 7758 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:31.783425 9112 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:31.993389 9157 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:31.993716 9157 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:31.995816 9157 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:31.996640 9097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:31.998313 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:31.998560 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:31.998656 9097 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:31.999047 9157 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:32.030108 9112 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:52663; Asking this server to re-register.
I20240503 02:13:32.030756 9113 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:51637; Asking this server to re-register.
I20240503 02:13:32.030840 8037 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:32.031495 7901 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:32.032554 9113 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:32.033324 9112 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:32.653096 9184 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:32.653363 9184 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:32.654419 9184 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:32.657161 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:32.657423 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:32.657564 9097 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:32.657855 9184 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:33.697296 9194 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:33.697525 9194 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:33.698202 9194 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:33.701357 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:33.701557 9097 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:33.701632 9097 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:33.701879 9194 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:33.959751 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 9078
I20240503 02:13:33.971551 9143 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:34.088971 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 9078
I20240503 02:13:35.009078 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--webserver_interface=127.4.17.126
--webserver_port=39555
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:35.386214 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:35.386574 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:35.386713 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:35.394773 9215 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:35.394918 9215 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:35.395001 9215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:35.395081 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:35.395155 9215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:35.395241 9215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:35.405192 9215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=39555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:35.406769 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:35.408792 9215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:35.421387 9223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:35.422322 9224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:35.423287 9226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:35.424679 9215 server_base.cc:1034] running on GCE node
I20240503 02:13:35.425637 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:35.427304 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:35.428587 9215 hybrid_clock.cc:648] HybridClock initialized: now 1714702415428509 us; error 51 us; skew 500 ppm
I20240503 02:13:35.429076 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:35.436442 9215 webserver.cc:460] Webserver started at http://127.4.17.126:39555/ using document root <none> and password file <none>
I20240503 02:13:35.437510 9215 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:35.437660 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:35.443811 9215 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:35.447371 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:35.449064 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:35.449245 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:35.450183 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:35.471345 9215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:35.472276 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:35.472568 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:35.492228 9215 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:42485
I20240503 02:13:35.492245 9283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:42485 every 8 connection(s)
I20240503 02:13:35.494470 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:13:35.498952 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 9215
I20240503 02:13:35.502959 9284 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:35.506636 9284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap starting.
I20240503 02:13:35.521796 9284 log.cc:824] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:35.562873 7901 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:35.565968 9284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=45 ignored=0} inserts{seen=29 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 02:13:35.566628 9284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap complete.
I20240503 02:13:35.573801 9284 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:35.577234 9284 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Initialized, Role: FOLLOWER
I20240503 02:13:35.578131 9284 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 8.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:35.579026 9293 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:35.579288 9293 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:35.580168 9284 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:35.599969 9307 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:35.600095 9307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: loading cluster ID for follower catalog manager: success
I20240503 02:13:35.603608 8037 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:35.606818 9248 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:53669; Asking this server to re-register.
I20240503 02:13:35.607798 8037 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:35.608683 9307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: acquiring CA information for follower catalog manager: success
I20240503 02:13:35.610391 9248 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:35.618163 9307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:35.619050 9215 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:35.627267 9215 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:35.634896 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:35.637183 9248 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:43677; Asking this server to re-register.
I20240503 02:13:35.637969 7758 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:35.639761 9248 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:35.949852 9248 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:36891; Asking this server to re-register.
I20240503 02:13:35.950582 7901 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:35.952525 9248 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:35.961673 9293 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:35.961962 9293 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:35.964727 9293 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:35.966203 9234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:35.968024 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:35.968299 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:35.968396 9234 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:35.968839 9293 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:36.044776 9100 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:36.045055 9100 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:36.045832 9100 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:36.046849 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:36.047024 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:36.047130 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:36.047382 9100 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:36.549537 9323 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:36.549813 9323 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:36.550747 9323 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:36.553229 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:36.553421 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:36.553519 9234 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:36.553805 9323 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:13:37.356838 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 11: this message will repeat every 5th retry.
I20240503 02:13:37.527031 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:37.527261 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:37.528106 9339 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:37.530897 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:37.531175 9234 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:37.531301 9234 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:37.531597 9339 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:38.000269 4165 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu pid 9215
I20240503 02:13:38.008105 9279 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:13:38.114320 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 11: this message will repeat every 5th retry.
I20240503 02:13:38.118737 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:38.119024 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:38.119752 9339 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
I20240503 02:13:38.120167 4165 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu with pid 9215
W20240503 02:13:38.142930 7926 connection.cc:535] client connection to 127.4.17.126:42485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:38.143337 7790 connection.cc:535] client connection to 127.4.17.126:42485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:38.143483 8047 connection.cc:535] client connection to 127.4.17.126:42485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:38.709388 4165 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
/tmp/dist-test-taskIVKwFe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--webserver_interface=127.4.17.126
--webserver_port=39555
--builtin_ntp_servers=127.4.17.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--hive_metastore_uris=thrift://127.0.0.1:42951
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 02:13:39.098181 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:39.098511 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:39.098603 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:39.107043 9366 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20240503 02:13:39.107168 9366 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20240503 02:13:39.107214 9366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:39.107275 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:39.107326 9366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:39.107386 9366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:39.117795 9366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.17.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42951
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.17.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.17.126
--webserver_port=39555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:39.119628 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:39.121686 9366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:39.135016 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:39.136202 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:39.137337 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:39.138502 9366 server_base.cc:1034] running on GCE node
I20240503 02:13:39.139468 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:39.141310 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:39.142604 9366 hybrid_clock.cc:648] HybridClock initialized: now 1714702419142510 us; error 71 us; skew 500 ppm
I20240503 02:13:39.143011 9366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:39.146468 9366 webserver.cc:460] Webserver started at http://127.4.17.126:39555/ using document root <none> and password file <none>
I20240503 02:13:39.147362 9366 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:39.147503 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:39.154454 9366 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:39.158298 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:39.160127 9366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:13:39.160347 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data,/tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
uuid: "f2855cc86e414615a261b7a7d4bd3293"
format_stamp: "Formatted at 2024-05-03 02:13:07 on dist-test-slave-ddxx"
I20240503 02:13:39.161381 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:39.188858 9366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:39.189953 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:39.190295 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:39.210891 9366 rpc_server.cc:307] RPC server started. Bound to: 127.4.17.126:42485
I20240503 02:13:39.210906 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.17.126:42485 every 8 connection(s)
I20240503 02:13:39.212646 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIVKwFe/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1714702331233159-4165-0/minicluster-data/master-0/data/info.pb
I20240503 02:13:39.219622 4165 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIVKwFe/build/asan/bin/kudu as pid 9366
I20240503 02:13:39.219987 9437 sys_catalog.cc:263] Verifying existing consensus state
I20240503 02:13:39.224694 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap starting.
I20240503 02:13:39.242753 7758 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:39.243465 9437 log.cc:824] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:39.282387 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=45 ignored=0} inserts{seen=29 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 02:13:39.283098 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: Bootstrap complete.
I20240503 02:13:39.291122 9437 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:39.294625 9437 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2855cc86e414615a261b7a7d4bd3293, State: Initialized, Role: FOLLOWER
I20240503 02:13:39.295372 9437 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 8.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:39.296195 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } } }
I20240503 02:13:39.296545 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:13:39.297416 9437 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:39.310410 9458 catalog_manager.cc:1259] Loaded cluster ID: 8f236d440dd14cddaa06325797d6ac04
I20240503 02:13:39.310662 9458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: loading cluster ID for follower catalog manager: success
I20240503 02:13:39.315797 9458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: acquiring CA information for follower catalog manager: success
I20240503 02:13:39.319288 9458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:13:39.347513 9401 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" instance_seqno: 1714702389470503) as {username='slave'} at 127.4.17.65:51331; Asking this server to re-register.
I20240503 02:13:39.348393 7758 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:39.350880 9401 ts_manager.cc:194] Registered new tserver with Master: 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901)
I20240503 02:13:39.354782 7901 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:39.356954 9401 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" instance_seqno: 1714702389991460) as {username='slave'} at 127.4.17.66:53087; Asking this server to re-register.
I20240503 02:13:39.357894 7901 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:39.360211 9401 ts_manager.cc:194] Registered new tserver with Master: c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
I20240503 02:13:39.357864 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:39.365912 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:39.654372 8037 heartbeater.cc:347] Connected to a master server at 127.4.17.126:42485
I20240503 02:13:39.656050 9401 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" instance_seqno: 1714702390505221) as {username='slave'} at 127.4.17.67:49387; Asking this server to re-register.
I20240503 02:13:39.656751 8037 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:39.658499 9401 ts_manager.cc:194] Registered new tserver with Master: c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095)
I20240503 02:13:39.732906 9444 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:39.733211 9444 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:39.735064 9444 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:39.735826 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:39.737486 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:39.737788 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:39.737929 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:39.738350 9444 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:40.123445 9350 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:40.123652 9350 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:40.124415 9350 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:40.125517 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:40.125684 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:40.125844 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:40.126096 9350 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:40.311273 9473 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:40.311496 9473 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:40.312264 9473 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:40.314936 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:40.315140 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:40.315256 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:40.315521 9473 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:40.866952 9487 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:40.867125 9487 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:40.867852 9487 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:40.877221 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:40.877458 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:40.877564 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:40.877851 9487 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:13:41.588531 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 11: this message will repeat every 5th retry.
I20240503 02:13:41.736271 4165 master-stress-test.cc:578] Killing test thread 1/25
I20240503 02:13:41.786929 9504 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:41.787138 9504 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:41.787817 9504 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:41.790474 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:41.790717 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:41.790830 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:41.791097 9504 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:42.620700 9521 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:42.620895 9521 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:42.621518 9521 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:42.624012 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:42.624363 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:42.624524 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:42.624817 9521 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:44.369493 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:44.369735 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:44.392853 9550 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:44.393117 9550 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:44.393998 9550 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:44.396057 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:44.396258 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:44.396375 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:44.396767 9550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:44.930841 9561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:44.931073 9561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:44.931807 9561 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:44.933110 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 02:13:44.937376 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:44.937609 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:44.937732 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:44.938016 9561 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:13:47.458067 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:48.181738 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 16: this message will repeat every 5th retry.
I20240503 02:13:49.372561 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:49.372763 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:50.903476 9652 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:50.903697 9652 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:50.904426 9652 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:50.905782 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:13:50.911233 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:50.911440 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:50.911563 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:50.911849 9652 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:13:51.158597 9641 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:51.158828 9641 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:51.159979 9641 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:51.160976 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:51.161135 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:51.161293 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:51.161593 9641 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:13:51.436300 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 16: this message will repeat every 5th retry.
I20240503 02:13:54.375891 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:54.376114 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:13:54.395062 9694 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:54.395292 9694 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:54.396005 9694 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:54.396979 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:54.397161 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:54.397274 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:54.397533 9694 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:54.817993 9710 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:54.818198 9710 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:13:54.818845 9710 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:13:54.820888 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:13:54.821162 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:13:54.821285 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:13:54.821565 9710 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:13:57.740545 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 21: this message will repeat every 5th retry.
I20240503 02:13:58.342100 9723 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:58.342320 9723 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:58.343084 9723 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:58.344270 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:13:58.345053 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:13:58.345204 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:58.345475 9723 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:58.347955 9723 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:58.348078 9723 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:13:58.348685 9723 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:13:58.349664 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:13:58.349834 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:13:58.350008 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:13:58.350283 9723 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:13:58.723762 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 21: this message will repeat every 5th retry.
I20240503 02:13:59.378962 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:13:59.379145 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:14:01.087062 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:04.382108 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:04.382282 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:05.186106 9856 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:05.186357 9856 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:14:05.187080 9856 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:14:05.188319 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:14:05.190052 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:14:05.190336 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:14:05.190466 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:14:05.190737 9856 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:14:07.933763 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:08.309403 9895 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:08.309659 9895 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:08.310588 9895 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:08.311668 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:14:08.311857 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:14:08.312009 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:08.312328 9895 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:14:08.483237 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:09.385124 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:09.385313 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:09.948022 9933 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:09.948237 9933 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:14:09.948982 9933 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:14:09.952776 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:09.952998 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:14:09.953100 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:14:09.953337 9933 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:14:10.913269 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:11.070526 7900 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (130 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:14:11.070562 7902 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (107 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:11.121707 8038 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (85 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:11.121711 8036 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (103 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:14:11.323076 7759 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (134 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:14:12.280336 9969 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:12.280519 9969 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:12.281205 9969 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:12.282002 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:12.282160 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:12.282307 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:12.282558 9969 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:14.388456 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:14.388656 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:16.642874 10019 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:16.643046 10019 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:16.643674 10019 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:16.644642 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:16.644824 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:16.644989 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:16.645236 10019 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:14:18.541810 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:18.828812 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:19.395447 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:19.395637 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:14:20.591500 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:22.779310 10100 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:22.779531 10100 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:22.780282 10100 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:22.781366 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:14:22.781538 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:14:22.781680 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:22.781976 10100 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:24.321475 10100 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:24.321686 10100 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:24.322513 10100 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:24.323462 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:24.323633 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:24.323779 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:24.324038 10100 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:24.398545 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:24.398759 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:27.901489 10182 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:27.901703 10182 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:14:27.902555 10182 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:14:27.903717 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:14:27.905054 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:27.905234 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:14:27.905340 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:14:27.905611 10182 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:14:28.636651 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:28.745551 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:29.401353 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:29.401551 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:14:30.236023 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:34.404264 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:34.404461 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:34.986368 10232 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:34.986588 10232 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:34.987321 10232 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:34.988291 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:34.988468 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:34.988598 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:34.988996 10232 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:14:38.560140 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:38.561450 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:39.407675 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:39.407877 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:14:40.183097 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:40.467561 10343 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:40.467869 10343 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:40.468818 10343 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:40.469941 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:14:40.470142 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:14:40.470242 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:40.470525 10343 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:42.545843 10343 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.546020 10343 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:42.546716 10343 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:42.547756 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:42.547941 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:42.548065 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:42.548343 10343 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:42.588649 10385 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.588902 10385 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:14:42.589713 10385 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:14:42.591053 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:14:42.592509 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:42.592711 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:14:42.592819 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:14:42.593125 10385 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:14:44.411121 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:44.411331 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:46.650027 10425 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.650216 10425 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:46.650915 10425 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:46.651926 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:46.652112 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:46.652208 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:46.652452 10425 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:14:48.472484 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:48.484220 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:49.414606 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:49.414799 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:14:49.931672 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:50.339135 10490 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:50.339347 10490 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:50.340128 10490 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:50.341084 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:14:50.341264 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:14:50.341377 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:50.341701 10490 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:54.417860 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:54.418037 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:14:57.960119 10574 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:57.960322 10574 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:14:57.961273 10574 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:14:57.962191 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:14:57.962333 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:14:57.962491 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:14:57.962721 10574 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:57.967906 10604 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:57.968079 10604 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:14:57.968907 10604 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:14:57.970157 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:14:57.971531 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:14:57.971736 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:14:57.971860 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:14:57.972123 10604 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:14:58.403601 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:58.690485 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 51: this message will repeat every 5th retry.
I20240503 02:14:59.421037 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:14:59.421227 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:00.251294 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:00.742132 10637 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:00.742352 10637 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:00.743191 10637 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:00.744252 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:00.744410 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:00.744575 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:00.744987 10637 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:04.424065 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:04.424264 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:15:04.474948 10680 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:04.475142 10680 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:04.475872 10680 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:04.476816 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:04.477041 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:04.477185 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:04.477444 10680 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:15:07.669214 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:08.164186 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 56: this message will repeat every 5th retry.
I20240503 02:15:09.427197 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:09.427431 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:11.093978 7900 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (276 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:15:11.123272 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:11.355374 8036 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (223 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:15:11.625080 7757 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (347 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:15:12.891503 10812 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:12.891744 10812 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:15:12.892550 10812 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:15:12.893774 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:15:12.895187 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:15:12.895344 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:15:12.895433 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:15:12.895663 10812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:15:13.037873 10786 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:13.038095 10786 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:13.038995 10786 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:13.039763 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:15:13.040027 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:15:13.040164 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:13.040457 10786 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:14.430470 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:14.430681 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:17.082321 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:17.902357 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:19.433951 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:19.434218 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:15:20.296025 10914 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:20.296213 10914 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:20.297029 10914 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:20.298022 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:15:20.298225 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:15:20.298405 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:20.300772 10914 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:15:20.732793 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:21.122910 10914 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:21.123101 10914 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:21.123816 10914 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:21.124861 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:21.125070 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:21.125160 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:21.125468 10914 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:24.437413 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:24.437608 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:15:25.954756 10945 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:25.954962 10945 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:25.955741 10945 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:25.956573 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:25.956739 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:25.956893 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:25.957170 10945 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:15:26.914789 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:28.571811 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 66: this message will repeat every 5th retry.
I20240503 02:15:29.440505 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:29.440716 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:30.320513 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 66: this message will repeat every 5th retry.
I20240503 02:15:30.472316 11066 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:30.472579 11066 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:15:30.473603 11066 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:15:30.474854 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:15:30.476249 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:15:30.476471 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:15:30.476580 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:15:30.476965 11066 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:15:30.905475 11013 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:30.905673 11013 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:30.906674 11013 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:30.907754 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:15:30.907994 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:15:30.908116 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:30.908401 11013 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:34.443764 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:34.443938 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:36.613451 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:38.192742 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 71: this message will repeat every 5th retry.
I20240503 02:15:39.386232 9391 maintenance_manager.cc:419] P f2855cc86e414615a261b7a7d4bd3293: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033358
I20240503 02:15:39.446858 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:39.447026 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:15:39.447782 9390 maintenance_manager.cc:643] P f2855cc86e414615a261b7a7d4bd3293: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.028s sys 0.000s Metrics: {"bytes_written":13724,"cfile_init":1,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":29,"thread_start_us":269,"threads_started":1,"wal-append.queue_time_us":379}
I20240503 02:15:39.449069 9391 maintenance_manager.cc:419] P f2855cc86e414615a261b7a7d4bd3293: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1427 bytes on disk
I20240503 02:15:39.449882 9390 maintenance_manager.cc:643] P f2855cc86e414615a261b7a7d4bd3293: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:15:40.648281 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 71: this message will repeat every 5th retry.
I20240503 02:15:43.364408 11266 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:43.364641 11266 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:15:43.365367 11266 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:15:43.366665 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:15:43.368103 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:15:43.368306 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:15:43.368427 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:15:43.368696 11266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:15:43.915184 11216 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:43.915377 11216 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:43.916119 11216 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:43.917200 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:43.917388 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:43.917521 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:43.917821 11216 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:44.449947 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:44.450163 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:46.736995 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:48.309353 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 76: this message will repeat every 5th retry.
I20240503 02:15:49.109232 11349 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:49.109443 11349 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:49.110476 11349 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:49.111704 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:15:49.111920 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:15:49.112051 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:49.112337 11349 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:49.453308 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:49.453514 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:50.820614 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 76: this message will repeat every 5th retry.
I20240503 02:15:54.456947 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:54.457194 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:15:56.889469 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:59.214399 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 81: this message will repeat every 5th retry.
I20240503 02:15:59.330686 11538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:59.330870 11538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:15:59.331743 11538 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:15:59.333070 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:15:59.335541 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:15:59.335808 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:15:59.335924 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:15:59.336261 11538 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:15:59.459836 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:15:59.460031 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:15:59.471050 11533 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:59.471249 11533 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:15:59.471933 11533 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:15:59.472951 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:15:59.473178 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:15:59.473385 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:15:59.473712 11533 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:00.862254 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:04.463145 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:04.463342 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:16:05.346210 11650 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:05.346397 11650 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:05.347203 11650 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:05.347966 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:05.348191 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:05.348284 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:05.348539 11650 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:06.869799 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:09.205111 11701 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:09.205312 11701 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:09.205996 11701 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:09.206718 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:09.206887 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:09.206983 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:09.207222 11701 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:16:09.466321 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:09.466529 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:09.561564 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:10.181816 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:11.286695 7900 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (427 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:16:11.558223 8038 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (325 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:16:11.911497 7759 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (503 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:16:13.435292 11799 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:13.435511 11799 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:16:13.436316 11799 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:16:13.437544 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:16:13.440730 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:16:13.440948 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:16:13.441066 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:16:13.441325 11799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:16:14.469671 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:14.469851 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:16:16.373427 11762 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:16.373627 11762 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:16.374406 11762 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:16.375375 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:16:16.375581 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:16:16.375690 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:16.375941 11762 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:16.903647 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:19.472841 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:19.473120 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:20.186733 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 91: this message will repeat every 5th retry.
W20240503 02:16:20.204051 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:24.475844 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:24.476050 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:16:25.534916 12025 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:25.535161 12025 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:25.536056 12025 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:25.537140 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:25.537338 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:25.537441 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:25.537721 12025 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:26.486904 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:29.479022 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:29.479205 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:29.634858 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:29.828526 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:33.335515 12157 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:33.335736 12157 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:16:33.336633 12157 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:16:33.338073 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:16:33.339453 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:16:33.339672 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:16:33.339778 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:16:33.340059 12157 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:16:34.024475 12168 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:34.024698 12168 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:34.025516 12168 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:34.026566 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:16:34.026737 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:16:34.026852 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:34.027115 12168 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:16:34.481938 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:34.482121 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:37.386277 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:37.973467 12237 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:37.973739 12237 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:37.974416 12237 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:37.975267 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:37.975430 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:37.975565 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:37.975854 12237 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:39.010231 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:39.485299 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:39.485476 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:39.766333 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:44.173178 12317 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:44.173404 12317 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:44.174477 12317 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:44.175482 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:44.175649 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:44.175784 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:44.176030 12317 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:16:44.488575 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:44.488783 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:16:45.703136 12317 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:45.703337 12317 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:45.704068 12317 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:45.705019 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:16:45.705224 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:16:45.705349 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:45.705569 12317 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:47.193004 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 106: this message will repeat every 5th retry.
W20240503 02:16:48.255975 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:49.491473 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:49.491676 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:49.888587 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:50.902777 12474 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:50.902997 12474 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:16:50.903846 12474 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:16:50.905107 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:16:50.906500 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:16:50.906652 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:16:50.906734 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:16:50.906982 12474 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:16:52.984259 12480 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:52.984464 12480 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:52.985255 12480 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:52.986476 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:16:52.986701 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:16:52.986877 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:52.987164 12480 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:16:54.494805 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:54.495023 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:16:54.908987 12550 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:54.909186 12550 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:16:54.909790 12550 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:16:54.911909 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:16:54.912125 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:16:54.912199 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:16:54.912500 12550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:16:57.082110 12560 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:57.082334 12560 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:16:57.083123 12560 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:16:57.084189 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:16:57.084748 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:16:57.084950 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:16:57.085258 12560 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:16:57.304162 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 111: this message will repeat every 5th retry.
W20240503 02:16:57.379129 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 111: this message will repeat every 5th retry.
I20240503 02:16:59.497797 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:16:59.497953 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:16:59.750276 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:00.086169 12639 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:00.086336 12639 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:17:00.086974 12639 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:17:00.088131 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:17:00.089349 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:00.089527 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:17:00.089614 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:17:00.089810 12639 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:17:01.597749 12669 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:01.597970 12669 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:17:01.598492 12669 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:17:01.600783 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:01.601004 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:17:01.601090 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:17:01.601316 12669 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:17:04.459633 12711 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:04.459836 12711 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:04.460815 12711 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:04.461843 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:17:04.462034 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:17:04.462162 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:04.462452 12711 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:04.500880 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:04.501134 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:06.029500 12711 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:06.029700 12711 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:06.030586 12711 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:06.031607 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:06.031787 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:06.031924 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:06.032222 12711 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:17:06.943367 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:07.127166 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:09.503803 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:09.503957 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:17:09.817442 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:11.435755 7900 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (579 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:17:11.760205 8038 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (445 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:12.153651 7759 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (691 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
I20240503 02:17:14.506780 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:14.506923 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:16.367053 12919 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:16.367241 12919 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:17:16.367967 12919 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:17:16.369161 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:17:16.370491 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:16.370656 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:17:16.370744 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:17:16.370956 12919 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:17:16.586609 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:17.183583 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:17.840970 12933 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:17.841200 12933 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:17.841935 12933 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:17.843050 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:17.843278 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:17.843436 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:17.843739 12933 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:17:19.211658 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:19.509696 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:19.509843 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:21.153993 13004 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:21.154203 13004 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:21.155052 13004 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:21.155946 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:21.156118 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:21.156267 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:21.156540 13004 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:23.508672 13038 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:23.508932 13038 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:23.509691 13038 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:23.510704 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:17:23.510957 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:17:23.511107 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:23.511371 13038 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:24.512745 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:24.512889 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:17:26.843178 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:27.232625 13113 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:27.232844 13113 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:17:27.233585 13113 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:17:27.234792 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:17:27.236047 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:27.236233 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:17:27.236312 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:17:27.236586 13113 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:17:27.367022 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:29.313400 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:29.515933 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:29.516098 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:30.839998 13150 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:30.840195 13150 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:30.840960 13150 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:30.841943 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:30.842139 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:30.842259 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:30.842512 13150 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:34.519161 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:34.519316 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:17:36.957070 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 131: this message will repeat every 5th retry.
W20240503 02:17:37.584358 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:38.408092 13304 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:38.408308 13304 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:38.409299 13304 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:38.410324 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:17:38.410521 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:17:38.410614 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:38.410943 13304 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:17:39.069554 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:39.522257 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:39.522414 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:44.525419 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:44.525614 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:45.926661 13437 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:45.926919 13437 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:45.927610 13437 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:45.928685 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:45.928889 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:45.929036 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:45.929312 13437 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:46.785529 13452 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:46.785748 13452 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:17:46.786573 13452 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:17:46.787667 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:17:46.788735 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:17:46.788892 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:17:46.789034 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:17:46.789238 13452 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:17:47.352502 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:47.835223 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:48.786960 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:49.528057 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:49.528200 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:50.431042 13511 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:50.431237 13511 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:50.431967 13511 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:50.432996 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:17:50.433166 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:17:50.433315 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:50.433568 13511 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:17:54.531016 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:54.531160 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:17:55.725574 13604 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:17:55.725785 13604 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:17:55.726604 13604 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:17:55.727465 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:17:55.727654 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:17:55.727771 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:17:55.727985 13604 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:17:57.035503 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:58.144876 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:58.631429 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:59.533907 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:17:59.534049 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:18:00.253317 13684 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:00.253517 13684 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:00.254294 13684 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:00.255271 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:18:00.255466 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:18:00.255635 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:00.255985 13684 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:18:01.968262 13718 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:01.968477 13718 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:18:01.969322 13718 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:18:01.970512 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:18:01.973944 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:18:01.974119 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:18:01.974231 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:18:01.974479 13718 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:18:04.536414 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:18:04.536587 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:18:06.516688 13794 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:06.516933 13794 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:06.517668 13794 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:06.518577 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:18:06.518743 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:18:06.518862 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:06.519124 13794 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:18:06.936753 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 146: this message will repeat every 5th retry.
W20240503 02:18:07.438774 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 146: this message will repeat every 5th retry.
I20240503 02:18:08.206256 13820 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:08.206414 13820 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:18:08.207012 13820 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:18:08.207860 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:18:08.209106 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:18:08.209225 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:18:08.209323 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:18:08.209537 13820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240503 02:18:08.310987 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 146: this message will repeat every 5th retry.
I20240503 02:18:09.539086 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:18:09.539235 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:18:10.734383 13858 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:10.734650 13858 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:10.735616 13858 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:10.736629 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:18:10.736802 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:18:10.737100 7926 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:10.737388 13858 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:18:10.916693 13858 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:10.916898 13858 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:10.917593 13858 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:10.918494 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:18:10.918711 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:18:10.918897 7926 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:10.919183 13858 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:18:11.912312 7902 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (709 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:18:11.962896 8038 heartbeater.cc:644] Failed to heartbeat to 127.4.17.125:45323 (565 consecutive failures): Network error: Failed to ping master at 127.4.17.125:45323: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:18:12.450731 7757 heartbeater.cc:644] Failed to heartbeat to 127.4.17.124:34141 (901 consecutive failures): Network error: Failed to ping master at 127.4.17.124:34141: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:18:14.542006 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:18:14.542176 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:18:16.760862 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 151: this message will repeat every 5th retry.
W20240503 02:18:16.917337 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 151: this message will repeat every 5th retry.
W20240503 02:18:17.403554 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 151: this message will repeat every 5th retry.
I20240503 02:18:19.544905 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:18:19.545106 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 02:18:19.760752 14008 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:19.761020 14008 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2855cc86e414615a261b7a7d4bd3293" member_type: VOTER last_known_addr { host: "127.4.17.126" port: 42485 } } peers { permanent_uuid: "96754aa0950c421b84eb59e6c6fbd2fe" member_type: VOTER last_known_addr { host: "127.4.17.125" port: 45323 } } peers { permanent_uuid: "0fefe1a587244be2b2dec28f7fa3f9d9" member_type: VOTER last_known_addr { host: "127.4.17.124" port: 34141 } }
I20240503 02:18:19.761735 14008 leader_election.cc:288] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323), 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141)
W20240503 02:18:19.762486 9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:18:19.763599 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 96754aa0950c421b84eb59e6c6fbd2fe (127.4.17.125:45323): Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111)
W20240503 02:18:19.763741 9387 leader_election.cc:334] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0fefe1a587244be2b2dec28f7fa3f9d9 (127.4.17.124:34141): Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
I20240503 02:18:19.763813 9387 leader_election.cc:302] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2855cc86e414615a261b7a7d4bd3293; no voters: 0fefe1a587244be2b2dec28f7fa3f9d9, 96754aa0950c421b84eb59e6c6fbd2fe
I20240503 02:18:19.764079 14008 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2855cc86e414615a261b7a7d4bd3293 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240503 02:18:20.460582 9391 maintenance_manager.cc:419] P f2855cc86e414615a261b7a7d4bd3293: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1427 bytes on disk
I20240503 02:18:20.466737 9390 maintenance_manager.cc:643] P f2855cc86e414615a261b7a7d4bd3293: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.006s sys 0.000s Metrics: {"dirs.queue_time_us":323,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":171,"thread_start_us":442,"threads_started":2,"wal-append.queue_time_us":329}
I20240503 02:18:23.345769 14046 raft_consensus.cc:482] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:23.345997 14046 raft_consensus.cc:504] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:23.346827 14046 leader_election.cc:288] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:23.347890 7926 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
W20240503 02:18:23.348119 7924 leader_election.cc:341] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: 366a898b86db46baa61ed152e98305bb
I20240503 02:18:23.348223 7924 leader_election.cc:302] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:23.348466 14046 raft_consensus.cc:2726] T 366a898b86db46baa61ed152e98305bb P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:18:24.174225 14046 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:24.174389 14046 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:24.174964 14046 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:24.175630 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:18:24.175804 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:18:24.175904 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:24.176132 14046 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:18:24.547984 9366 master_runner.cc:186] Error getting master registration for 127.4.17.125:45323: Network error: Client connection negotiation failed: client connection to 127.4.17.125:45323: connect: Connection refused (error 111),
I20240503 02:18:24.548163 9366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 02:18:26.246177 7648 consensus_peers.cc:485] T fb1acfaf1d3d4673b89f0de854e60d0e P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb1acfaf1d3d4673b89f0de854e60d0e. This is attempt 156: this message will repeat every 5th retry.
W20240503 02:18:26.476559 7791 consensus_peers.cc:485] T f3014429e2a54d01ba806f9b26e58485 P c9f0228ec6c64da5b5d84fa3ebe0ade1 -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3014429e2a54d01ba806f9b26e58485. This is attempt 156: this message will repeat every 5th retry.
W20240503 02:18:27.255702 7648 consensus_peers.cc:485] T d818c02e5f8c4bc8816a5cc702cf0b78 P 6f6a6a27cc814e0a86b1767df5f5a31b -> Peer c67f5cc3ea7a4529a01ede7d54fe4710 (127.4.17.67:41095): Couldn't send request to peer c67f5cc3ea7a4529a01ede7d54fe4710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d818c02e5f8c4bc8816a5cc702cf0b78. This is attempt 156: this message will repeat every 5th retry.
I20240503 02:18:27.578223 14114 raft_consensus.cc:482] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:27.578400 14114 raft_consensus.cc:504] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67f5cc3ea7a4529a01ede7d54fe4710" member_type: VOTER last_known_addr { host: "127.4.17.67" port: 41095 } } peers { permanent_uuid: "6f6a6a27cc814e0a86b1767df5f5a31b" member_type: VOTER last_known_addr { host: "127.4.17.65" port: 45901 } } peers { permanent_uuid: "c9f0228ec6c64da5b5d84fa3ebe0ade1" member_type: VOTER last_known_addr { host: "127.4.17.66" port: 33615 } }
I20240503 02:18:27.579224 14114 leader_election.cc:288] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901), c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615)
W20240503 02:18:27.580271 7926 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f6a6a27cc814e0a86b1767df5f5a31b (127.4.17.65:45901): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
W20240503 02:18:27.580452 7924 leader_election.cc:341] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f0228ec6c64da5b5d84fa3ebe0ade1 (127.4.17.66:33615): Not found: Tablet not found: a3907eb584054a73973b35efc23a5ce8
I20240503 02:18:27.580574 7924 leader_election.cc:302] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c67f5cc3ea7a4529a01ede7d54fe4710; no voters: 6f6a6a27cc814e0a86b1767df5f5a31b, c9f0228ec6c64da5b5d84fa3ebe0ade1
I20240503 02:18:27.580819 14114 raft_consensus.cc:2726] T a3907eb584054a73973b35efc23a5ce8 P c67f5cc3ea7a4529a01ede7d54fe4710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:18:28.522471 8047 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:18:28.522785 8047 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:18:28.522920 8047 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
W20240503 02:18:28.523015 8047 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
F20240503 02:18:28.535020 8075 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03d6519166457e8a6519e2133ad38f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535377 8066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Aborted at 1714702708 (unix time) try "date -d @1714702708" if you are using GNU date ***
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03d6519166457e8a6519e2133ad38f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535377 8066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03d6519166457e8a6519e2133ad38f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535377 8066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03d6519166457e8a6519e2133ad38f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535377 8066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.536571 8079 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03d6519166457e8a6519e2133ad38f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535377 8066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.536571 8079 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.536787 8054 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a15ad42e445d4011a7ce7a358eb31e66 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240503 02:18:28.535207 8070 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.4.17.126:42485,127.4.17.125:45323,127.4.17.124:34141, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.4.17.124:34141: connect: Connection refused (error 111)F20240503 02:18:28.535331 8058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f03